公开(公告)号：US10812823B2

公开(公告)日：2020-10-20

申请号：US16032925

申请日：2018-07-11

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Guy Côté ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/436 ,  H04N19/577 ,  H04N19/196 ,  H04N19/139 ,  H04N19/176 ,  H04N19/105 ,  H04N19/147

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and performs a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding system also includes a main encoding pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a global motion vector based on data from the low resolution motion estimation block. The main encoding pipeline may utilize the global motion vector in determining a candidate inter prediction mode.

公开(公告)号：US20200021842A1

公开(公告)日：2020-01-16

申请号：US16032968

申请日：2018-07-11

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/176 ,  H04N19/52 ,  H04N19/567 ,  H04N19/186

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data, performs a motion estimation search to determine motion vector candidates that are each indicative of location of a downscaled reference sample, and prunes one or more of the motion vector candidates. The video encoding system also includes a main pipeline that determines encoding parameters to be used to encode image data based at least partially the remaining motion vector candidates.

公开(公告)号：US09571846B2

公开(公告)日：2017-02-14

申请号：US14039764

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Timothy John Millet ,  Mark P. Rygh ,  Craig M. Okruhlica ,  Jim C. Chou ,  Guy Cote ,  Gaurav S. Gulati ,  Joseph J. Cheng ,  Joseph P. Bratt

IPC: H04N19/423 ,  H04N19/56 ,  H04N19/53

CPC classification number: H04N19/423 ,  H04N19/53

Abstract: Block processing pipeline methods and apparatus in which reference data are stored to a memory according to tile formats to reduce memory accesses when fetching the data from the memory. When the pipeline stores reference data from a current frame being processed to memory as a reference frame, the reference samples are stored in macroblock sequential order. Each macroblock sample set is stored as a tile. Reference data may be stored in tile formats for luma and chroma. Chroma reference data may be stored in tile formats for chroma 4:2:0, 4:2:2, and/or 4:4:4 formats. A stage of the pipeline may write luma and chroma reference data for macroblocks to memory according to one or more of the macroblock tile formats in a modified knight's order. The stage may delay writing the reference data from the macroblocks until the macroblocks have been fully processed by the pipeline.

Abstract translation: 块处理管道方法和装置，其中参考数据根据瓦片格式存储到存储器中，以在从存储器取出数据时减少存储器访问。 当流水线将正在处理的当前帧的参考数据存储为参考帧时，参考样本以宏块顺序存储。 每个宏块样本集被存储为一个图块。 参考数据可以以瓦片和色度的瓦片格式存储。 色度参考数据可以以瓦4：2：0,4：2：2和/或4：4：4格式的瓦片格式存储。 流水线的一个阶段可以根据改进的骑士顺序中的一个或多个宏块瓦片格式将宏块的亮度和色度参考数据写入存储器。 该阶段可以延迟从宏块写入参考数据，直到宏块已被管道完全处理。

公开(公告)号：US11336915B2

公开(公告)日：2022-05-17

申请号：US17066451

申请日：2020-10-08

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Guy Côté ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/436 ,  H04N19/577 ,  H04N19/196 ,  H04N19/139 ,  H04N19/176 ,  H04N19/105 ,  H04N19/147

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and performs a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding system also includes a main encoding pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a global motion vector based on data from the low resolution motion estimation block. The main encoding pipeline may utilize the global motion vector in determining a candidate inter prediction mode.

公开(公告)号：US10757437B2

公开(公告)日：2020-08-25

申请号：US14334614

申请日：2014-07-17

Applicant: Apple Inc.

Inventor： Jim C. Chou ,  Mark P. Rygh

IPC: H04N19/513 ,  H04N19/433 ,  H04N19/53 ,  H04N19/523 ,  H04N19/533

Abstract: Block processing pipeline methods and apparatus in which. motion estimation is performed at a stage of a motion estimation module for a current block with respect to a reference frame at one or more partition sizes to determine candidate motion vectors. The candidate motion vectors may be passed to a next stage for refinement. Motion estimation may then be performed at the next stage to refine the motion vectors. In performing motion estimation at this stage, the input motion vectors of at least one partition size received from the previous stage may be used as candidate motion vectors in searches for at least one other partition size.

公开(公告)号：US09336558B2

公开(公告)日：2016-05-10

申请号：US14039845

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Guy Cote ,  Timothy John Millet ,  Joseph J. Cheng ,  Mark P. Rygh ,  Jim C. Chou

IPC: H04N19/436 ,  G06T1/20 ,  H04N19/176 ,  G06T1/60 ,  H04N19/423

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/176 ,  H04N19/40 ,  H04N19/423 ,  H04N19/436 ,  H04N19/91

Abstract: In the video encoders described herein, blocks of pixels from a video frame may be encoded (e.g., using CAVLC encoding) in a block processing pipeline using wavefront ordering (e.g., in knight's order). Each of the encoded blocks may be written to a particular one of multiple DMA buffers such that the encoded blocks written to each of the buffers represent consecutive blocks of the video frame in scan order. A transcode pipeline may operate in parallel with (or at least overlapping) the operation of the block processing pipeline. The transcode pipeline may read encoded blocks from the buffers in scan order and merge them into a single bit stream (in scan order). A transcoder core of the transcode pipeline may decode the encoded blocks and encode them using a different encoding process (e.g., CABAC). In some cases, the transcoder may be bypassed.

Abstract translation: 在本文描述的视频编码器中，来自视频帧的像素块可以使用波前排序（例如骑士顺序）在块处理流水线中进行编码（例如，使用CAVLC编码）。 每个编码块可以被写入多个DMA缓冲器中的特定一个，使得写入每个缓冲器的编码块以扫描顺序表示视频帧的连续块。 代码流水线可以与（或至少重叠）块处理流水线的操作并行操作。 代码流水线可以以扫描顺序从缓冲器读取编码块，并将它们合并成单个位流（按扫描顺序）。 代码转换流水线的代码转换器核心可以解码编码的块，并使用不同的编码过程（例如，CABAC）对它们进行编码。 在某些情况下，代码转换器可能被旁路。

公开(公告)号：US20160021385A1

公开(公告)日：2016-01-21

申请号：US14334614

申请日：2014-07-17

Applicant: Apple Inc.

Inventor： Jim C. Chou ,  Mark P. Rygh

IPC: H04N19/513

CPC classification number: H04N19/513 ,  H04N19/433 ,  H04N19/523 ,  H04N19/53 ,  H04N19/533

Abstract: Block processing pipeline methods and apparatus in which. motion estimation is performed at a stage of a motion estimation module for a current block with respect to a reference frame at one or more partition sizes to determine candidate motion vectors. The candidate motion vectors may be passed to a next stage for refinement. Motion estimation may then be performed at the next stage to refine the motion vectors. In performing motion estimation at this stage, the input motion vectors of at least one partition size received from the previous stage may be used as candidate motion vectors in searches for at least one other partition size.

Abstract translation: 块处理管道的方法和装置。 在针对当前块的运动估计模块的阶段，对于一个或多个分区大小的参考帧执行运动估计，以确定候选运动矢量。 可以将候选运动矢量传递到下一阶段以进行细化。 然后可以在下一阶段执行运动估计，以细化运动矢量。 在这一阶段进行运动估计时，从至少一个分区大小的输入运动向量可以被用作搜索至少一个其他分区大小的候选运动矢量。

公开(公告)号：US20150091920A1

公开(公告)日：2015-04-02

申请号：US14039804

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Mark P. Rygh ,  Guy Cote ,  Timothy John Millet ,  Joseph J. Cheng

IPC: G06T1/20

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/176 ,  H04N19/423 ,  H04N19/436 ,  H04N19/523 ,  H04N19/593

Abstract: Memory latency tolerance methods and apparatus for maintaining an overall level of performance in block processing pipelines that prefetch reference data into a search window. In a general memory latency tolerance method, search window processing in the pipeline may be monitored. If status of search window processing changes in a way that affects pipeline throughput, then pipeline processing may be modified. The modification may be performed according to no stall methods, stall recovery methods, and/or stall prevention methods. In no stall methods, a block may be processed using the data present in the search window without waiting for the missing reference data. In stall recovery methods, the pipeline is allowed to stall, and processing is modified for subsequent blocks to speed up the pipeline and catch up in throughput. In stall prevention methods, processing is adjusted in advance of the pipeline encountering a stall condition.

Abstract translation: 存储器延迟容限方法和装置，用于在预处理参考数据到搜索窗口的块处理管线中维持整体性能水平。 在通用存储器延迟容限方法中，可以监视流水线中的搜索窗口处理。 如果搜索窗口处理的状态以影响流水线吞吐量的方式改变，则可以修改流水线处理。 修改可以根据没有失速方法，失速恢复方法和/或失速预防方法进行。 在没有停止方法的情况下，可以使用搜索窗口中存在的数据来处理块，而不用等待丢失的参考数据。 在失速恢复方法中，允许管道停止，并且修改后续块的处理以加速管道并追赶吞吐量。 在失速预防方法中，在遇到失速状况的管道之前调整处理。

公开(公告)号：US11006143B2

公开(公告)日：2021-05-11

申请号：US16032968

申请日：2018-07-11

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/176 ,  H04N19/567 ,  H04N19/186 ,  H04N19/52 ,  H04N19/124

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data, performs a motion estimation search to determine motion vector candidates that are each indicative of location of a downscaled reference sample, and prunes one or more of the motion vector candidates. The video encoding system also includes a main pipeline that determines encoding parameters to be used to encode image data based at least partially the remaining motion vector candidates.

公开(公告)号：US20170094293A1

公开(公告)日：2017-03-30

申请号：US14871848

申请日：2015-09-30

Applicant: Apple Inc.

Inventor： Jim C. Chou ,  Mark P. Rygh ,  Guy Côté

IPC: H04N19/42 ,  H04N19/103 ,  H04N19/182 ,  H04N19/186 ,  H04N19/139 ,  H04N19/147

CPC classification number: H04N19/439 ,  H04N19/109 ,  H04N19/139 ,  H04N19/147 ,  H04N19/156 ,  H04N19/157 ,  H04N19/17 ,  H04N19/182 ,  H04N19/186

Abstract: System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a motion estimation setup block, which dynamically adjusts a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline and select an initial candidate inter-frame prediction mode based at least on the setup configuration, a full-pel motion estimation block, which determines an intermediate candidate inter-frame prediction mode by performing a motion estimation search based on the initial candidate inter-frame prediction mode, a sub-pel motion estimation block, which determines a final candidate inter-frame prediction by performing a motion estimation search based on the intermediate candidate inter-frame prediction mode, and a mode decision block, which determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and determines a prediction mode used to prediction encoding the image data.