公开(公告)号：US09571846B2

公开(公告)日：2017-02-14

申请号：US14039764

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Timothy John Millet ,  Mark P. Rygh ,  Craig M. Okruhlica ,  Jim C. Chou ,  Guy Cote ,  Gaurav S. Gulati ,  Joseph J. Cheng ,  Joseph P. Bratt

IPC: H04N19/423 ,  H04N19/56 ,  H04N19/53

CPC classification number: H04N19/423 ,  H04N19/53

Abstract: Block processing pipeline methods and apparatus in which reference data are stored to a memory according to tile formats to reduce memory accesses when fetching the data from the memory. When the pipeline stores reference data from a current frame being processed to memory as a reference frame, the reference samples are stored in macroblock sequential order. Each macroblock sample set is stored as a tile. Reference data may be stored in tile formats for luma and chroma. Chroma reference data may be stored in tile formats for chroma 4:2:0, 4:2:2, and/or 4:4:4 formats. A stage of the pipeline may write luma and chroma reference data for macroblocks to memory according to one or more of the macroblock tile formats in a modified knight's order. The stage may delay writing the reference data from the macroblocks until the macroblocks have been fully processed by the pipeline.

Abstract translation: 块处理管道方法和装置，其中参考数据根据瓦片格式存储到存储器中，以在从存储器取出数据时减少存储器访问。 当流水线将正在处理的当前帧的参考数据存储为参考帧时，参考样本以宏块顺序存储。 每个宏块样本集被存储为一个图块。 参考数据可以以瓦片和色度的瓦片格式存储。 色度参考数据可以以瓦4：2：0,4：2：2和/或4：4：4格式的瓦片格式存储。 流水线的一个阶段可以根据改进的骑士顺序中的一个或多个宏块瓦片格式将宏块的亮度和色度参考数据写入存储器。 该阶段可以延迟从宏块写入参考数据，直到宏块已被管道完全处理。

公开(公告)号：US20160065969A1

公开(公告)日：2016-03-03

申请号：US14474114

申请日：2014-08-30

Applicant: Apple Inc.

Inventor： Joseph J. Cheng ,  Timothy J. Millet ,  Shun Wai Go ,  Guy Cote

IPC: H04N19/176 ,  H04N19/182 ,  H04N19/46 ,  H04N19/172

CPC classification number: H04N19/176 ,  G06F9/461 ,  G06F9/4818 ,  H04N19/127 ,  H04N19/129 ,  H04N19/172 ,  H04N19/174 ,  H04N19/182 ,  H04N19/40 ,  H04N19/42 ,  H04N19/46

Abstract: A context switching method for video encoders that enables higher priority video streams to interrupt lower priority video streams. A high priority frame may be received for processing while another frame is being processed. The pipeline may be signaled to perform a context stop for the current frame. The pipeline stops processing the current frame at an appropriate place, and propagates the stop through the stages of the pipeline and to a transcoder through DMA. The stopping location is recorded. The video encoder may then process the higher-priority frame. When done, a context restart is performed and the pipeline resumes processing the lower-priority frame beginning at the recorded location. The transcoder may process data for the interrupted frame while the higher-priority frame is being processed in the pipeline, and similarly the pipeline may begin processing the lower-priority frame after the context restart while the transcoder completes processing the higher-priority frame.

Abstract translation: 一种视频编码器的上下文切换方法，其使得较高优先级的视频流能够中断较低优先级的视频流。 当处理另一帧时，可以接收高优先级帧以进行处理。 可以用信号通知流水线以执行当前帧的上下文停止。 流水线在适当的位置停止处理当前帧，并通过流水线传播停止点，并通过DMA传播到代码转换器。 记录停止位置。 视频编码器然后可以处理较高优先级的帧。 完成后，执行上下文重新启动，并且流水线从记录的位置恢复处理较低优先级的帧。 代码转换器可以在流水线中处理较高优先级帧的同时处理中断帧的数据，类似地，当代码转换器完成处理较高优先级帧时，流水线可以在上下文重新启动之后开始处理较低优先级的帧。

公开(公告)号：US10313683B2

公开(公告)日：2019-06-04

申请号：US14474114

申请日：2014-08-30

Applicant: Apple Inc.

Inventor： Joseph J. Cheng ,  Timothy J. Millet ,  Shun Wai Go ,  Guy Cote

IPC: H04N19/172 ,  H04N19/176 ,  H04N19/46 ,  G06F9/46 ,  G06F9/48 ,  H04N19/129 ,  H04N19/127 ,  H04N19/174 ,  H04N19/42 ,  H04N19/40 ,  H04N19/182

Abstract: A context switching method for video encoders that enables higher priority video streams to interrupt lower priority video streams. A high priority frame may be received for processing while another frame is being processed. The pipeline may be signaled to perform a context stop for the current frame. The pipeline stops processing the current frame at an appropriate place, and propagates the stop through the stages of the pipeline and to a transcoder through DMA. The stopping location is recorded. The video encoder may then process the higher-priority frame. When done, a context restart is performed and the pipeline resumes processing the lower-priority frame beginning at the recorded location. The transcoder may process data for the interrupted frame while the higher-priority frame is being processed in the pipeline, and similarly the pipeline may begin processing the lower-priority frame after the context restart while the transcoder completes processing the higher-priority frame.

公开(公告)号：US09336558B2

公开(公告)日：2016-05-10

申请号：US14039845

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Guy Cote ,  Timothy John Millet ,  Joseph J. Cheng ,  Mark P. Rygh ,  Jim C. Chou

IPC: H04N19/436 ,  G06T1/20 ,  H04N19/176 ,  G06T1/60 ,  H04N19/423

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/176 ,  H04N19/40 ,  H04N19/423 ,  H04N19/436 ,  H04N19/91

Abstract: In the video encoders described herein, blocks of pixels from a video frame may be encoded (e.g., using CAVLC encoding) in a block processing pipeline using wavefront ordering (e.g., in knight's order). Each of the encoded blocks may be written to a particular one of multiple DMA buffers such that the encoded blocks written to each of the buffers represent consecutive blocks of the video frame in scan order. A transcode pipeline may operate in parallel with (or at least overlapping) the operation of the block processing pipeline. The transcode pipeline may read encoded blocks from the buffers in scan order and merge them into a single bit stream (in scan order). A transcoder core of the transcode pipeline may decode the encoded blocks and encode them using a different encoding process (e.g., CABAC). In some cases, the transcoder may be bypassed.

Abstract translation: 在本文描述的视频编码器中，来自视频帧的像素块可以使用波前排序（例如骑士顺序）在块处理流水线中进行编码（例如，使用CAVLC编码）。 每个编码块可以被写入多个DMA缓冲器中的特定一个，使得写入每个缓冲器的编码块以扫描顺序表示视频帧的连续块。 代码流水线可以与（或至少重叠）块处理流水线的操作并行操作。 代码流水线可以以扫描顺序从缓冲器读取编码块，并将它们合并成单个位流（按扫描顺序）。 代码转换流水线的代码转换器核心可以解码编码的块，并使用不同的编码过程（例如，CABAC）对它们进行编码。 在某些情况下，代码转换器可能被旁路。

公开(公告)号：US20150091920A1

公开(公告)日：2015-04-02

申请号：US14039804

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： Mark P. Rygh ,  Guy Cote ,  Timothy John Millet ,  Joseph J. Cheng

IPC: G06T1/20

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/176 ,  H04N19/423 ,  H04N19/436 ,  H04N19/523 ,  H04N19/593

Abstract: Memory latency tolerance methods and apparatus for maintaining an overall level of performance in block processing pipelines that prefetch reference data into a search window. In a general memory latency tolerance method, search window processing in the pipeline may be monitored. If status of search window processing changes in a way that affects pipeline throughput, then pipeline processing may be modified. The modification may be performed according to no stall methods, stall recovery methods, and/or stall prevention methods. In no stall methods, a block may be processed using the data present in the search window without waiting for the missing reference data. In stall recovery methods, the pipeline is allowed to stall, and processing is modified for subsequent blocks to speed up the pipeline and catch up in throughput. In stall prevention methods, processing is adjusted in advance of the pipeline encountering a stall condition.

Abstract translation: 存储器延迟容限方法和装置，用于在预处理参考数据到搜索窗口的块处理管线中维持整体性能水平。 在通用存储器延迟容限方法中，可以监视流水线中的搜索窗口处理。 如果搜索窗口处理的状态以影响流水线吞吐量的方式改变，则可以修改流水线处理。 修改可以根据没有失速方法，失速恢复方法和/或失速预防方法进行。 在没有停止方法的情况下，可以使用搜索窗口中存在的数据来处理块，而不用等待丢失的参考数据。 在失速恢复方法中，允许管道停止，并且修改后续块的处理以加速管道并追赶吞吐量。 在失速预防方法中，在遇到失速状况的管道之前调整处理。

公开(公告)号：US20150084968A1

公开(公告)日：2015-03-26

申请号：US14037313

申请日：2013-09-25

Applicant: Apple Inc.

Inventor： Joseph J. Cheng ,  Guy Cote ,  Marc A. Schaub ,  Jim C. Chou

IPC: G06T1/20 ,  G06T1/60

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: Methods and apparatus for caching neighbor data in a block processing pipeline that processes blocks in knight's order with quadrow constraints. Stages of the pipeline may maintain two local buffers that contain data from neighbor blocks of a current block. A first buffer contains data from the last C blocks processed at the stage. A second buffer contains data from neighbor blocks on the last row of a previous quadrow. Data for blocks on the bottom row of a quadrow are stored to an external memory at the end of the pipeline. When a block on the top row of a quadrow is input to the pipeline, neighbor data from the bottom row of the previous quadrow is read from the external memory and passed down the pipeline, each stage storing the data in its second buffer and using the neighbor data in the second buffer when processing the block.

Abstract translation: 用于在块处理管道中缓存邻居数据的方法和装置，其以四限制约束以骑士顺序处理块。 管道的阶段可以维护两个包含当前块的相邻块的数据的本地缓冲器。 第一个缓冲区包含在该阶段处理的最后一个C块的数据。 第二个缓冲区包含来自前一个四边形最后一行的相邻块的数据。 四边形底行中的块的数据存储在流水线末端的外部存储器中。 当四边形的顶行上的块被输入到流水线时，从外部存储器读取来自前一个四边形的底行的邻居数据，并将其传送到流水线，每个级将数据存储在其第二缓冲器中，并使用 处理块时第二个缓冲区中的邻居数据。

公开(公告)号：US09305325B2

公开(公告)日：2016-04-05

申请号：US14037313

申请日：2013-09-25

Applicant: Apple Inc.

Inventor： Joseph J. Cheng ,  Guy Cote ,  Marc A. Schaub ,  Jim C. Chou

IPC: G06T1/20 ,  G06T1/60 ,  H04N19/61 ,  H04N19/423 ,  H04N19/436

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: Methods and apparatus for caching neighbor data in a block processing pipeline that processes blocks in knight's order with quadrow constraints. Stages of the pipeline may maintain two local buffers that contain data from neighbor blocks of a current block. A first buffer contains data from the last C blocks processed at the stage. A second buffer contains data from neighbor blocks on the last row of a previous quadrow. Data for blocks on the bottom row of a quadrow are stored to an external memory at the end of the pipeline. When a block on the top row of a quadrow is input to the pipeline, neighbor data from the bottom row of the previous quadrow is read from the external memory and passed down the pipeline, each stage storing the data in its second buffer and using the neighbor data in the second buffer when processing the block.

Abstract translation: 用于在块处理管道中缓存邻居数据的方法和装置，其以四限制约束以骑士顺序处理块。 管道的阶段可以维护两个包含当前块的相邻块的数据的本地缓冲器。 第一个缓冲区包含在该阶段处理的最后一个C块的数据。 第二个缓冲区包含来自前一个四边形最后一行的相邻块的数据。 四边形底行中的块的数据存储在流水线末端的外部存储器中。 当四边形的顶行上的块被输入到流水线时，从外部存储器读取来自前一个四边形的底行的邻居数据，并将其传送到流水线，每个级将数据存储在其第二缓冲器中，并使用 处理块时第二个缓冲区中的邻居数据。

公开(公告)号：US09292899B2

公开(公告)日：2016-03-22

申请号：US14037318

申请日：2013-09-25

Applicant: Apple Inc.

Inventor： Marc A. Schaub ,  Joseph J. Cheng ,  Mark P. Rygh ,  Guy Cote

IPC: G06F13/00 ,  G06T1/20 ,  G06T1/60 ,  H04N19/61 ,  H04N19/423 ,  H04N19/436

CPC classification number: G06T1/20 ,  G06T1/60 ,  H04N19/423 ,  H04N19/436 ,  H04N19/61

Abstract: Block processing pipeline methods and apparatus in which pixel data from a reference frame is prefetched into a search window memory. The search window may include two or more overlapping regions of pixels from the reference frame corresponding to blocks from the rows in the input frame that are currently being processed in the pipeline. Thus, the pipeline may process blocks from multiple rows of an input frame using one set of pixel data from a reference frame that is stored in a shared search window memory. The search window may be advanced by one column of blocks by initiating a prefetch for a next column of reference data from a memory. The pipeline may also include a reference data cache that may be used to cache a portion of a reference frame and from which at least a portion of a prefetch for the search window may be satisfied.

Abstract translation: 块处理流水线方法和装置，其中来自参考帧的像素数据被预取到搜索窗口存储器中。 搜索窗口可以包括对应于当前正在流水线处理的输入帧中的行的来自参考帧的两个或更多个重叠区域的像素。 因此，流水线可以使用来自存储在共享搜索窗口存储器中的参考帧的一组像素数据来处理来自输入帧的多行的块。 搜索窗口可以由一列块提前，通过从存储器发起下一列参考数据的预取。 流水线还可以包括可用于缓存参考帧的一部分的参考数据高速缓存，并且可以从该参考数据高速缓冲存储器可以满足搜索窗口的预取的至少一部分。

公开(公告)号：US20160065973A1

公开(公告)日：2016-03-03

申请号：US14472119

申请日：2014-08-28

Applicant: APPLE INC.

Inventor： Guy Cote ,  Joseph P. Bratt ,  Timothy J. Millet ,  Shing I. Kong ,  Joseph J. Cheng

IPC: H04N19/186 ,  H04N19/423 ,  H04N19/172 ,  H04N19/176

CPC classification number: H04N19/186 ,  G06F12/00 ,  G06F12/0207 ,  G06F12/0862 ,  G06F12/121 ,  G06F2212/1024 ,  G06F2212/455 ,  G06F2212/6024 ,  G06F2212/6026 ,  G06T1/60 ,  H04N19/127 ,  H04N19/172 ,  H04N19/176 ,  H04N19/423 ,  H04N19/433 ,  H04N19/439

Abstract: Methods and apparatus for caching reference data in a block processing pipeline. A cache may be implemented to which reference data corresponding to motion vectors for blocks being processed in the pipeline may be prefetched from memory. Prefetches for the motion vectors may be initiated one or more stages prior to a processing stage. Cache tags for the cache may be defined by the motion vectors. When a motion vector is received, the tags can be checked to determine if there are cache block(s) corresponding to the vector (cache hits) in the cache. Upon a cache miss, a cache block in the cache is selected according to a replacement policy, the respective tag is updated, and a prefetch (e.g., via DMA) for the respective reference data is issued.

Abstract translation: 在块处理流水线中缓存参考数据的方法和装置。 可以实现缓存，其可以从存储器预取哪个对应于在流水线中处理的块的运动矢量的参考数据。 可以在处理阶段之前一个或多个阶段启动用于运动矢量的预取。 高速缓存的缓存标签可以由运动向量定义。 当接收到运动矢量时，可以检查标签以确定是否存在与缓存中的向量（高速缓存命中）相对应的高速缓存块。 在缓存未命中时，根据替换策略来选择高速缓存中的高速缓存块，相应的标签被更新，并且发出用于各个参考数据的预取（例如，经由DMA）。

公开(公告)号：US20150092854A1

公开(公告)日：2015-04-02

申请号：US14039729

申请日：2013-09-27

Applicant: Apple Inc.

Inventor： James E. Orr ,  Timothy John Millet ,  Joseph J. Cheng ,  Nitin Bhargava ,  Guy Cote

IPC: H04N19/436 ,  H04N19/433 ,  H04N19/51 ,  H04N19/583 ,  H04N19/43

CPC classification number: H04N19/43 ,  G06T1/20 ,  H04N19/433 ,  H04N19/436 ,  H04N19/51 ,  H04N19/513

Abstract: A block processing pipeline that includes a software pipeline and a hardware pipeline that run in parallel. The software pipeline runs at least one block ahead of the hardware pipeline. The stages of the pipeline may each include a hardware pipeline component that performs one or more operations on a current block at the stage. At least one stage of the pipeline may also include a software pipeline component that determines a configuration for the hardware component at the stage of the pipeline for processing a next block while the hardware component is processing the current block. The software pipeline component may determine the configuration according to information related to the next block obtained from an upstream stage of the pipeline. The software pipeline component may also obtain and use information related to a block that was previously processed at the stage.

Abstract translation: 一个块处理流水线，包括一个软件流水线和并行运行的硬件流水线。 软件管道在硬件管道之前至少运行一个程序段。 流水线的各个阶段可以各自包括在该阶段对当前块执行一个或多个操作的硬件流水线组件。 管道的至少一个阶段还可以包括软件流水线组件，该软件流水线组件在硬件组件正在处理当前块时，在流水线阶段确定用于处理下一个块的硬件组件的配置。 软件管线组件可以根据从流水线的上游级获得的与下一块相关的信息来确定配置。 软件管道组件还可以获得并使用与先前在该阶段处理的块相关的信息。