公开(公告)号：US09218639B2

公开(公告)日：2015-12-22

申请号：US14039820

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Guy Cote ,  Mark P. Rygh ,  Timothy John Millet ,  Jim C. Chou ,  Joseph J. Cheng

IPC: G06T15/00 ,  G06T1/20 ,  H04N19/61 ,  H04N19/423 ,  H04N19/436

CPC classification number: G06T1/20 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: A knight's order processing method for block processing pipelines in which the next block input to the pipeline is taken from the row below and one or more columns to the left in the frame. The knight's order method may provide spacing between adjacent blocks in the pipeline to facilitate feedback of data from a downstream stage to an upstream stage. The rows of blocks in the input frame may be divided into sets of rows that constrain the knight's order method to maintain locality of neighbor block data. Invalid blocks may be input to the pipeline at the left of the first set of rows and at the right of the last set of rows, and the sets of rows may be treated as if they are horizontally arranged rather than vertically arranged, to maintain continuity of the knight's order algorithm.

Abstract translation: 一种用于块处理管线的骑士订单处理方法，其中从管线的下一个块输入下一个块，并且在该帧中从左侧获取一个或多个列。 骑士的订单方法可以在管道中的相邻块之间提供间隔，以便于数据从下游阶段到上游阶段的反馈。 输入帧中的块行可以被划分为限制骑士命令方法以维持相邻块数据的位置的行的集合。 无效的块可以被输入到第一组行的左侧和最后一组行的右侧的流水线，并且这些行的集合可以被视为水平排列而不是垂直排列，以保持连续性 的骑士秩序算法。

公开(公告)号：US20150255047A1

公开(公告)日：2015-09-10

申请号：US14201421

申请日：2014-03-07

Applicant: Apple Inc.

Inventor： Peter F. Holland ,  Guy Cote ,  Mark P. Rygh

IPC: G09G5/39 ,  G09G5/02

CPC classification number: G09G5/39 ,  G09G5/001 ,  G09G5/022 ,  G09G5/363 ,  G09G2340/02 ,  G09G2340/06 ,  G09G2340/10 ,  G09G2340/12 ,  G09G2360/121 ,  G09G2360/128 ,  G09G2360/16

Abstract: In an embodiment, a system includes a display processing unit configured to process a video sequence for a target display. In some embodiments, the display processing unit is configured to composite the frames from frames of the video sequence and one or more other image sources. The display processing unit may be configured to write the processed/composited frames to memory, and may also be configured to generate statistics over the frame data, where the generated statistics are usable to encode the frame in a video encoder. The display processing unit may be configured to write the generated statistics to memory, and the video encoder may be configured to read the statistics and the frames. The video encoder may be configured to encode the frame responsive to the statistics.

Abstract translation: 在一个实施例中，系统包括被配置为处理用于目标显示的视频序列的显示处理单元。 在一些实施例中，显示处理单元被配置为从视频序列的帧和一个或多个其它图像源合成帧。 显示处理单元可以被配置为将经处理/合成的帧写入存储器，并且还可以被配置为生成关于帧数据的统计信息，其中生成的统计信息可用于对视频编码器中的帧进行编码。 显示处理单元可以被配置为将生成的统计信息写入存储器，并且视频编码器可以被配置为读取统计信息和帧。 视频编码器可以被配置为响应于统计信息对帧进行编码。

公开(公告)号：US10477233B2

公开(公告)日：2019-11-12

申请号：US14871778

申请日：2015-09-30

Applicant: Apple Inc.

Inventor： Jim C. Chou ,  Mark P. Rygh ,  Guy Côté

IPC: H04N19/52 ,  H04N19/583 ,  H04N19/59 ,  H04N19/567 ,  H04N19/587 ,  H04N19/573 ,  H04N19/61 ,  H04N19/533 ,  H04N19/523 ,  H04N19/156 ,  H04N19/57

Abstract: System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a mode decision block, which selects a first inter-frame prediction mode used to prediction encode a first prediction unit, and a motion estimation block, which receives the first inter-frame prediction mode as feedback from the mode decision block when processing a second prediction unit; determines an initial candidate inter-frame prediction mode of the second prediction unit based at least in part on the first inter-frame prediction mode; and determines a final candidate inter-frame prediction mode of the second prediction unit by performing a first motion estimation search based at least in part on the initial candidate inter-frame prediction mode. The mode decision block determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and a prediction mode used to prediction encode the second prediction unit based at least in part on the rate-distortion cost.

公开(公告)号：US10225572B2

公开(公告)日：2019-03-05

申请号：US14871848

申请日：2015-09-30

Applicant: Apple Inc.

Inventor： Jim C. Chou ,  Mark P. Rygh ,  Guy Côté

IPC: H04N7/12 ,  H04N11/02 ,  H04N11/04 ,  H04N19/42 ,  H04N19/139 ,  H04N19/147 ,  H04N19/182 ,  H04N19/186 ,  H04N19/109 ,  H04N19/156 ,  H04N19/157 ,  H04N19/17

Abstract: System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a motion estimation setup block, which dynamically adjusts a setup configuration of the motion estimation block based at least in part on operational parameters of the video encoding pipeline and select an initial candidate inter-frame prediction mode based at least on the setup configuration, a full-pel motion estimation block, which determines an intermediate candidate inter-frame prediction mode by performing a motion estimation search based on the initial candidate inter-frame prediction mode, a sub-pel motion estimation block, which determines a final candidate inter-frame prediction by performing a motion estimation search based on the intermediate candidate inter-frame prediction mode, and a mode decision block, which determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and determines a prediction mode used to prediction encoding the image data.

公开(公告)号：US09472168B2

公开(公告)日：2016-10-18

申请号：US14201421

申请日：2014-03-07

Applicant: Apple Inc.

Inventor： Peter F. Holland ,  Guy Cote ,  Mark P. Rygh

IPC: G09G5/39 ,  G09G5/00 ,  G09G5/36 ,  G09G5/02

CPC classification number: G09G5/39 ,  G09G5/001 ,  G09G5/022 ,  G09G5/363 ,  G09G2340/02 ,  G09G2340/06 ,  G09G2340/10 ,  G09G2340/12 ,  G09G2360/121 ,  G09G2360/128 ,  G09G2360/16

Abstract: In an embodiment, a system includes a display processing unit configured to process a video sequence for a target display. In some embodiments, the display processing unit is configured to composite the frames from frames of the video sequence and one or more other image sources. The display processing unit may be configured to write the processed/composited frames to memory, and may also be configured to generate statistics over the frame data, where the generated statistics are usable to encode the frame in a video encoder. The display processing unit may be configured to write the generated statistics to memory, and the video encoder may be configured to read the statistics and the frames. The video encoder may be configured to encode the frame responsive to the statistics.

Abstract translation: 在一个实施例中，系统包括被配置为处理用于目标显示的视频序列的显示处理单元。 在一些实施例中，显示处理单元被配置为从视频序列的帧和一个或多个其它图像源合成帧。 显示处理单元可以被配置为将经处理/合成的帧写入存储器，并且还可以被配置为生成关于帧数据的统计信息，其中生成的统计信息可用于对视频编码器中的帧进行编码。 显示处理单元可以被配置为将生成的统计信息写入存储器，并且视频编码器可以被配置为读取统计信息和帧。 视频编码器可以被配置为响应于统计信息对帧进行编码。

公开(公告)号：US09224186B2

公开(公告)日：2015-12-29

申请号：US14039804

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Mark P. Rygh ,  Guy Cote ,  Timothy John Millet ,  Joseph J. Cheng

IPC: G06T1/20 ,  G06T1/60 ,  H04N19/423 ,  H04N19/436 ,  H04N19/176

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/176 ,  H04N19/423 ,  H04N19/436 ,  H04N19/523 ,  H04N19/593

Abstract: Memory latency tolerance methods and apparatus for maintaining an overall level of performance in block processing pipelines that prefetch reference data into a search window. In a general memory latency tolerance method, search window processing in the pipeline may be monitored. If status of search window processing changes in a way that affects pipeline throughput, then pipeline processing may be modified. The modification may be performed according to no stall methods, stall recovery methods, and/or stall prevention methods. In no stall methods, a block may be processed using the data present in the search window without waiting for the missing reference data. In stall recovery methods, the pipeline is allowed to stall, and processing is modified for subsequent blocks to speed up the pipeline and catch up in throughput. In stall prevention methods, processing is adjusted in advance of the pipeline encountering a stall condition.

Abstract translation: 存储器延迟容限方法和装置，用于在预处理参考数据到搜索窗口的块处理管线中维持整体性能水平。 在通用存储器延迟容限方法中，可以监视流水线中的搜索窗口处理。 如果搜索窗口处理的状态以影响流水线吞吐量的方式改变，则可以修改流水线处理。 修改可以根据没有失速方法，失速恢复方法和/或失速预防方法进行。 在没有停止方法的情况下，可以使用搜索窗口中存在的数据来处理块，而不用等待丢失的参考数据。 在失速恢复方法中，允许管道停止，并且修改后续块的处理以加速管道并追赶吞吐量。 在失速预防方法中，在遇到失速状况的管道之前调整处理。

公开(公告)号：US20150091914A1

公开(公告)日：2015-04-02

申请号：US14039820

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Guy Cote ,  Mark P. Rygh ,  Timothy John Millet ,  Jim C. Chou ,  Joseph J. Cheng

IPC: G06T1/20

CPC classification number: G06T1/20 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: A knight's order processing method for block processing pipelines in which the next block input to the pipeline is taken from the row below and one or more columns to the left in the frame. The knight's order method may provide spacing between adjacent blocks in the pipeline to facilitate feedback of data from a downstream stage to an upstream stage. The rows of blocks in the input frame may be divided into sets of rows that constrain the knight's order method to maintain locality of neighbor block data. Invalid blocks may be input to the pipeline at the left of the first set of rows and at the right of the last set of rows, and the sets of rows may be treated as if they are horizontally arranged rather than vertically arranged, to maintain continuity of the knight's order algorithm.

Abstract translation: 一种用于块处理管线的骑士订单处理方法，其中从管线的下一个块输入下一个块，并且在该帧中从左侧获取一个或多个列。 骑士的订单方法可以在管道中的相邻块之间提供间隔，以便于数据从下游阶段到上游阶段的反馈。 输入帧中的块行可以被划分为限制骑士命令方法以维持相邻块数据的位置的行的集合。 无效的块可以被输入到第一组行的左侧和最后一组行的右侧的流水线，并且这些行的集合可以被视为水平排列而不是垂直排列，以保持连续性 的骑士秩序算法。

公开(公告)号：US20150084970A1

公开(公告)日：2015-03-26

申请号：US14037318

申请日：2013-09-25

Applicant: Apple Inc.

Inventor： Marc A. Schaub ,  Joseph J. Cheng ,  Mark P. Rygh ,  Guy Cote

IPC: G06T1/20 ,  G06T1/60

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: Block processing pipeline methods and apparatus in which pixel data from a reference frame is prefetched into a search window memory. The search window may include two or more overlapping regions of pixels from the reference frame corresponding to blocks from the rows in the input frame that are currently being processed in the pipeline. Thus, the pipeline may process blocks from multiple rows of an input frame using one set of pixel data from a reference frame that is stored in a shared search window memory. The search window may be advanced by one column of blocks by initiating a prefetch for a next column of reference data from a memory. The pipeline may also include a reference data cache that may be used to cache a portion of a reference frame and from which at least a portion of a prefetch for the search window may be satisfied.

Abstract translation: 块处理流水线方法和装置，其中来自参考帧的像素数据被预取到搜索窗口存储器中。 搜索窗口可以包括对应于当前正在流水线处理的输入帧中的行的来自参考帧的两个或更多个重叠区域的像素。 因此，流水线可以使用来自存储在共享搜索窗口存储器中的参考帧的一组像素数据来处理来自输入帧的多行的块。 搜索窗口可以由一列块提前，通过从存储器发起下一列参考数据的预取。 流水线还可以包括可用于缓存参考帧的一部分的参考数据高速缓存，并且可以从该参考数据高速缓冲存储器可以满足搜索窗口的预取的至少一部分。

公开(公告)号：US20210029376A1

公开(公告)日：2021-01-28

申请号：US17066451

申请日：2020-10-08

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Guy Côté ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/105 ,  H04N19/139 ,  H04N19/196 ,  H04N19/436 ,  H04N19/577 ,  H04N19/176

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and performs a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding system also includes a main encoding pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a global motion vector based on data from the low resolution motion estimation block. The main encoding pipeline may utilize the global motion vector in determining a candidate inter prediction mode.

公开(公告)号：US20200021841A1

公开(公告)日：2020-01-16

申请号：US16032925

申请日：2018-07-11

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Guy Côté ,  Mark P. Rygh

IPC: H04N19/527 ,  H04N19/436 ,  H04N19/577 ,  H04N19/105 ,  H04N19/196 ,  H04N19/139 ,  H04N19/176

Abstract: Systems and methods for improving operational efficiency of a video encoding system used to encode image data are provided. In embodiments, the video encoding system includes a low resolution pipeline that includes a low resolution motion estimation block, which generates downscaled image data by reducing resolution of the image data and performs a motion estimation search using the downscaled image data and previously downscaled image data. The video encoding system also includes a main encoding pipeline in parallel with the low resolution pipeline that includes a motion estimation block, which determines a global motion vector based on data from the low resolution motion estimation block. The main encoding pipeline may utilize the global motion vector in determining a candidate inter prediction mode.