公开(公告)号：US20130329809A1

公开(公告)日：2013-12-12

申请号：US13631719

申请日：2012-09-28

Applicant: APPLE INC.

Inventor： Athanasios Leontaris ,  Haitao Guo ,  Xiaojin Shi

IPC: H04N7/26

CPC classification number: H04N21/44209 ,  H04N19/107 ,  H04N19/164 ,  H04N19/172 ,  H04N19/196 ,  H04N19/65 ,  H04N19/89 ,  H04N21/6379

Abstract: An error recovery method may be engaged by an encoder to recover from misalignment between reference picture caches at the encoder and decoder. When a communication error is detected between a coder and a decoder, a number of non-acknowledged reference frames present in the decoder's reference picture cache may be estimated. Thereafter, frames may be coded as reference frames in a number greater or equal to the number of non-acknowledged reference frames that are estimated to be present in the decoder's reference picture cache. Thereafter, ordinary coding operations may resume. Typically, a final reference frame that is coded in the error recovery mode will be coded as a synchronization frame that has high coding quality. The coded reference frames that precede it may be coded at low quality (or may be coded as SKIP-coded frames). On reception and decoding, the preceding frames may cause the decoder to flush from its reference picture cache any non-acknowledged reference frames that otherwise might collide with the new synchronization frame. In this manner, alignment between the encoder and decoder may be restored.

Abstract translation: 编码器可以接合错误恢复方法以从编码器和解码器处的参考图像高速缓存之间的未对准中恢复。 当在编码器和解码器之间检测到通信错误时，可以估计存在于解码器的参考图像高速缓存中的多个未确认参考帧。 此后，帧可以被编码为大于或等于估计存在于解码器的参考图像缓存中的未确认参考帧的数量的参考帧。 此后，可以恢复普通编码操作。 通常，以错误恢复模式编码的最终参考帧将被编码为具有高编码质量的同步帧。 其之前的编码参考帧可以以低质量编码（或者可以被编码为SKIP编码的帧）。 在接收和解码时，前述帧可能导致解码器从其参考图像缓存中刷新否则可能与新的同步帧相冲突的任何未确认的参考帧。 以这种方式，可以恢复编码器和解码器之间的对准。

公开(公告)号：US20240112303A1

公开(公告)日：2024-04-04

申请号：US18371872

申请日：2023-09-22

Applicant: Apple Inc.

Inventor： Vincent Chapdelaine-Couture ,  Emmanuel Piuze-Phaneuf ,  Julien Monat Rodier ,  Hermannus J. Damveld ,  Xiaojin Shi ,  Sebastian Gaweda

IPC: G06T5/00 ,  G06T5/50 ,  G06T7/246

CPC classification number: G06T5/001 ,  G06T5/50 ,  G06T7/248

Abstract: In some implementations, a method includes: obtaining image data associated with a physical environment; obtaining first contextual information including at least one of first user information associated with a current state of a user of the computing system, first application information associated with a first application being executed by the computing system, and first environment information associated with a current state of the physical environment; selecting a first set of perspective correction operations based at least in part on the first contextual information; generating first corrected image data by performing the first set of perspective correction operations on the image data; and causing presentation of the first corrected image data.

公开(公告)号：US11561621B2

公开(公告)日：2023-01-24

申请号：US16600830

申请日：2019-10-14

Applicant: Apple Inc.

Inventor： Feng Tang ,  Chong Chen ,  Haitao Guo ,  Xiaojin Shi ,  Thorsten Gernoth

IPC: G06F3/01 ,  G06F3/03 ,  G06K9/62 ,  G06F3/16 ,  G06V40/10

Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.

公开(公告)号：US10664963B1

公开(公告)日：2020-05-26

申请号：US16032879

申请日：2018-07-11

Applicant: Apple Inc.

Inventor： Francesco Rossi ,  Xiaohuan C. Wang ,  Bartlomiej W. Rymkowski ,  Xiaojin Shi ,  Marco Zuliani ,  Alexey Marinichev

IPC: G06T5/50 ,  G06N3/08 ,  G06N3/04

Abstract: Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.

公开(公告)号：US20200042096A1

公开(公告)日：2020-02-06

申请号：US16600830

申请日：2019-10-14

Applicant: Apple Inc.

Inventor： Feng Tang ,  Chong Chen ,  Haitao Guo ,  Xiaojin Shi ,  Thorsten Gernoth

IPC: G06F3/01 ,  G06F3/03 ,  G06F3/16 ,  G06K9/00

Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.

公开(公告)号：US09811721B2

公开(公告)日：2017-11-07

申请号：US14706649

申请日：2015-05-07

Applicant: Apple Inc.

Inventor： Feng Tang ,  Ang Li ,  Xiaojin Shi

IPC: G06K9/00 ,  H04N13/02 ,  G06K9/46 ,  G06F3/01 ,  G06F3/03 ,  G06F3/042 ,  G06T7/246 ,  G06T7/254 ,  H04N13/00

CPC classification number: G06K9/00355 ,  G06F3/017 ,  G06F3/0304 ,  G06F3/0425 ,  G06K9/4609 ,  G06T7/246 ,  G06T7/254 ,  G06T2200/04 ,  G06T2207/30196 ,  H04N13/207 ,  H04N13/271 ,  H04N2013/0085

Abstract: In the field of Human-computer interaction (HCI), i.e., the study of the interfaces between people (i.e., users) and computers, understanding the intentions and desires of how the user wishes to interact with the computer is a very important problem. The ability to understand human gestures, and, in particular, hand gestures, as they relate to HCI, is a very important aspect in understanding the intentions and desires of the user in a wide variety of applications. In this disclosure, a novel system and method for three-dimensional hand tracking using depth sequences is described. Some of the major contributions of the hand tracking system described herein include: 1.) a robust hand detector that is invariant to scene background changes; 2.) a bi-directional tracking algorithm that prevents detected hands from always drifting closer to the front of the scene (i.e., forward along the z-axis of the scene); and 3.) various hand verification heuristics.

公开(公告)号：US20170090584A1

公开(公告)日：2017-03-30

申请号：US14865850

申请日：2015-09-25

Applicant: Apple Inc.

Inventor： Feng Tang ,  Chong Chen ,  Haitao Guo ,  Xiaojin Shi ,  Thorsten Gernoth

IPC: G06F3/01 ,  G06K9/00 ,  G06F3/03 ,  G06F3/16 ,  G06T7/00 ,  G06T7/20

CPC classification number: G06F3/017 ,  G06F3/012 ,  G06F3/0304 ,  G06F3/16 ,  G06K9/00375 ,  G06K9/00389 ,  G06K9/6269 ,  G06K9/6282 ,  G06K2209/40

Abstract: Varying embodiments of intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene, such as walls, furniture, and humans may be evaluated and monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent or desire as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, for example, expressed through fine hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, for example, optical or non-optical type depth sensors. The depth information may be interpreted in “slices” (three-dimensional regions of space having a relatively small depth) until one or more candidate hand structures are detected. Once detected, each candidate hand structure may be confirmed or rejected based on its own unique physical properties (e.g., shape, size and continuity to an arm structure). Each confirmed hand structure may be submitted to a depth-aware filtering process before its own unique three-dimensional features are quantified into a high-dimensional feature vector. A two-step classification scheme may be applied to the feature vectors to identify a candidate gesture (step 1), and to reject candidate gestures that do not meet a gesture-specific identification operation (step-2). The identified gesture may be used to initiate some action controlled by a computer system.

公开(公告)号：US09380312B2

公开(公告)日：2016-06-28

申请号：US14331091

申请日：2014-07-14

Applicant: Apple Inc.

Inventor： Guy Cote ,  Xiaojin Shi

IPC: G06K9/00 ,  G06K9/36 ,  H04N19/196 ,  G06K9/46 ,  G06T1/20 ,  H04N19/124 ,  H04N19/139 ,  H04N19/176

CPC classification number: H04N19/196 ,  G06K9/325 ,  G06K9/4642 ,  G06T1/20 ,  H04N19/124 ,  H04N19/139 ,  H04N19/14 ,  H04N19/176 ,  H04N19/42

Abstract: A block input component of a video encoding pipeline may, for a block of pixels in a video frame, compute gradients in multiple directions, and may accumulate counts of the computed gradients in one or more histograms. The block input component may analyze the histogram(s) to compute block-level statistics and determine whether a dominant gradient direction exists in the block, indicating the likelihood that it represents an image containing text. If text is likely, various encoding parameter values may be selected to improve the quality of encoding for the block (e.g., by lowering a quantization parameter value). The computed statistics or selected encoding parameter values may be passed to other stages of the pipeline, and used to bias or control selection of a prediction mode, an encoding mode, or a motion vector. Frame-level or slice-level parameter values may be generated from gradient histograms of multiple blocks.

Abstract translation: 对于视频帧中的像素块，视频编码流水线的块输入分量可以在多个方向上计算梯度，并且可以在一个或多个直方图中累积计算的梯度的计数。 块输入组件可以分析直方图以计算块级统计，并确定块中是否存在显性梯度方向，指示它代表包含文本的图像的可能性。 如果文本很可能，则可以选择各种编码参数值以提高块的编码质量（例如，通过降低量化参数值）。 所计算的统计量或所选择的编码参数值可以被传递到流水线的其他阶段，并且用于偏置或控制预测模式，编码模式或运动矢量的选择。 可以从多个块的梯度直方图生成帧级或片级参数值。

公开(公告)号：US09332309B2

公开(公告)日：2016-05-03

申请号：US13631719

申请日：2012-09-28

Applicant: Apple Inc.

Inventor： Athanasios Leontaris ,  Haitao Guo ,  Xiaojin Shi

IPC: H04N19/65 ,  H04N19/89 ,  H04N21/442 ,  H04N21/6379 ,  H04N19/172 ,  H04N19/196 ,  H04N19/107 ,  H04N19/164

CPC classification number: H04N21/44209 ,  H04N19/107 ,  H04N19/164 ,  H04N19/172 ,  H04N19/196 ,  H04N19/65 ,  H04N19/89 ,  H04N21/6379

Abstract: An error recovery method may be engaged by an encoder to recover from misalignment between reference picture caches at the encoder and decoder. When a communication error is detected between a coder and a decoder, a number of non-acknowledged reference frames present in the decoder's reference picture cache may be estimated. Thereafter, frames may be coded as reference frames in a number greater or equal to the number of non-acknowledged reference frames that are estimated to be present in the decoder's reference picture cache. Thereafter, ordinary coding operations may resume. Typically, a final reference frame that is coded in the error recovery mode will be coded as a synchronization frame that has high coding quality. The coded reference frames that precede it may be coded at low quality (or may be coded as SKIP-coded frames). On reception and decoding, the preceding frames may cause the decoder to flush from its reference picture cache any non-acknowledged reference frames that otherwise might collide with the new synchronization frame. In this manner, alignment between the encoder and decoder may be restored.

Abstract translation: 编码器可以接合错误恢复方法以从编码器和解码器处的参考图像高速缓存之间的未对准中恢复。 当在编码器和解码器之间检测到通信错误时，可以估计存在于解码器的参考图像高速缓存中的多个未确认参考帧。 此后，帧可以被编码为大于或等于估计存在于解码器的参考图像缓存中的未确认参考帧的数量的参考帧。 此后，可以恢复普通编码操作。 通常，以错误恢复模式编码的最终参考帧将被编码为具有高编码质量的同步帧。 其之前的编码参考帧可以以低质量编码（或者可以被编码为SKIP编码的帧）。 在接收和解码时，前述帧可能导致解码器从其参考图像缓存中刷新否则可能与新的同步帧相冲突的任何未确认的参考帧。 以这种方式，可以恢复编码器和解码器之间的对准。

公开(公告)号：US09313488B2

公开(公告)日：2016-04-12

申请号：US14175555

申请日：2014-02-07

Applicant: Apple Inc.

Inventor： Hsi-Jung Wu ,  Barin Geoffry Haskell ,  Xiaojin Shi ,  James Oliver Normile

IPC: H04N7/12 ,  H04N19/44 ,  H04N19/149 ,  H04N19/152

CPC classification number: H04N19/44 ,  H04N19/149 ,  H04N19/152

Abstract: Disclosed is a system and method of controlling a video decoder, including a reviewing channel data representing coded video data generated by an encoder to identify parameters of a hypothetical reference decoder (HRD) used by the encoder during coding operations. A parameter representing an exit data rate requirement of a coded picture buffer (CPB) of the HRD is compared against exit rate performance of the video decoder. If the exit rate performance of the video coder matches the exit rate requirement of the HRD, the coded video data is decoded, otherwise, a certain decoding degradation scheme can be applied, including disabling decoder from decoding the coded video data.