公开(公告)号：US09514525B2

公开(公告)日：2016-12-06

申请号：US14449011

申请日：2014-07-31

Applicant: Apple Inc.

Inventor： Suk Hwan Lim ,  D. Amnon Silverstein

IPC: H04N5/217 ,  G06T5/20

CPC classification number: G06T5/20 ,  G06T5/002 ,  G06T5/50 ,  G06T2207/20024 ,  G06T2207/20182 ,  G06T2207/20201 ,  H04N5/217

Abstract: A temporal filter in an image processing pipeline may perform filtering using spatial filtering and noise history. A given pixel of a current image frame may be received for filtering at a temporal filter. A filtering weight may be determined for blending the given pixel with a corresponding pixel of a reference image frame that was previously filtered at the temporal filter. The filtering weight may be determined based on neighboring pixels of the given pixel in the current image frame and corresponding pixels in the reference image frame. The filtering weight may be adjusted according to a quality score indicating noise history for the corresponding pixel in the reference image frame. Based on the filtering weight, a filtered version of the given pixel may be generated, blending the given pixel and the corresponding pixel to store as part of a filtered version of the current image frame.

Abstract translation: 图像处理流水线中的时间滤波器可以使用空间滤波和噪声历史来执行滤波。 可以接收当前图像帧的给定像素以在时间滤波器处进行滤波。 可以确定用于将给定像素与先前在时间滤波器处被滤波的参考图像帧的对应像素混合的滤波权重。 可以基于当前图像帧中的给定像素和参考图像帧中的相应像素的相邻像素来确定滤波权重。 可以根据指示参考图像帧中的相应像素的噪声历史的质量得分来调整滤波权重。 基于过滤权重，可以生成给定像素的滤波版本，将给定像素和相应像素混合存储为当前图像帧的滤波版本的一部分。

公开(公告)号：US09462189B2

公开(公告)日：2016-10-04

申请号：US14449024

申请日：2014-07-31

Applicant: Apple Inc.

Inventor： Christopher L. Mills ,  David R. Pope ,  D. Amnon Silverstein

IPC: H04N5/232 ,  G06T3/00

CPC classification number: H04N5/2329 ,  G06T3/00 ,  H04N5/23254 ,  H04N5/23258 ,  H04N5/23267

Abstract: An image signal processor of a device, apparatus, or computing system that includes a camera capable of capturing image data may apply piecewise perspective transformations to image data received from the camera's image sensor. A scaling unit of an Image Signal Processor (ISP) may perform piecewise perspective transformations on a captured image to correct for rolling shutter artifacts and to provide video image stabilization. Image data may be divided into a series of horizontal slices and perspective transformations may be applied to each slice. The transformations may be based on motion data determined in any of various manners, such as by using gyroscopic data and/or optical-flow calculations. The piecewise perspective transforms may be encoded as Digital Difference Analyzer (DDA) steppers and may be implemented using separable scalar operations. The image signal processor may not write the received image data to system memory until after the transformations have been performed.

Abstract translation: 包括能够拍摄图像数据的相机的设备，装置或计算系统的图像信号处理器可以对从相机的图像传感器接收的图像数据应用分段透视变换。 图像信号处理器（ISP）的缩放单元可以对捕获的图像执行分段透视变换，以校正滚动快门伪影并提供视频图像稳定。 图像数据可以被划分为一系列水平切片，并且透视变换可以应用于每个切片。 变换可以基于以各种方式中的任何一种确定的运动数据，例如通过使用陀螺仪数据和/或光流计算。 分段透视变换可以被编码为数字差分分析器（DDA）步进器，并且可以使用可分离的标量运算来实现。 图像信号处理器可能不会将接收的图像数据写入系统存储器直到转换完成。

公开(公告)号：US09344626B2

公开(公告)日：2016-05-17

申请号：US14082390

申请日：2013-11-18

Applicant: Apple Inc.

Inventor： D. Amnon Silverstein ,  Shun Wai Go ,  Suk Hwan Lim ,  Timothy J. Millet ,  Ting Chen ,  Bin Ni

IPC: H04N7/01 ,  H04N5/232 ,  H04N9/87 ,  H04N1/21

CPC classification number: H04N5/23245 ,  H04N1/212 ,  H04N5/23216 ,  H04N5/23232 ,  H04N5/23293 ,  H04N7/0122 ,  H04N9/87

Abstract: In an embodiment, an electronic device may be configured to capture still frames during video capture, but may capture the still frames in the 4×3 aspect ratio and at higher resolution than the 16×9 aspect ratio video frames. The device may interleave high resolution, 4×3 frames and lower resolution 16×9 frames in the video sequence, and may capture the nearest higher resolution, 4×3 frame when the user indicates the capture of a still frame. Alternatively, the device may display 16×9 frames in the video sequence, and then expand to 4×3 frames when a shutter button is pressed. The device may capture the still frame and return to the 16×9 video frames responsive to a release of the shutter button.

Abstract translation: 在一个实施例中，电子设备可以被配置为在视频捕获期间捕获静止帧，但是可以以比4×3宽高比和高于16×9宽高比视频帧的分辨率捕获静止帧。 该装置可以在视频序列中交错高分辨率，4×3帧和较低分辨率的16×9帧，并且当用户指示拍摄静止帧时可以捕获最近的较高分辨率，4×3帧。 或者，设备可以在视频序列中显示16×9帧，然后当按下快门按钮时，扩展为4×3帧。 该装置可以捕获静止帧并响应于快门按钮的释放而返回到16×9视频帧。

公开(公告)号：US20160037073A1

公开(公告)日：2016-02-04

申请号：US14449024

申请日：2014-07-31

Applicant: Apple Inc.

Inventor： Christopher L. Mills ,  David R. Pope ,  D. Amnon Silverstein

IPC: H04N5/232

CPC classification number: H04N5/2329 ,  G06T3/00 ,  H04N5/23254 ,  H04N5/23258 ,  H04N5/23267

Abstract: An image signal processor of a device, apparatus, or computing system that includes a camera capable of capturing image data may apply piecewise perspective transformations to image data received from the camera's image sensor. A scaling unit of an Image Signal Processor (ISP) may perform piecewise perspective transformations on a captured image to correct for rolling shutter artifacts and to provide video image stabilization. Image data may be divided into a series of horizontal slices and perspective transformations may be applied to each slice. The transformations may be based on motion data determined in any of various manners, such as by using gyroscopic data and/or optical-flow calculations. The piecewise perspective transforms may be encoded as Digital Difference Analyzer (DDA) steppers and may be implemented using separable scalar operations. The image signal processor may not write the received image data to system memory until after the transformations have been performed.

Abstract translation: 包括能够拍摄图像数据的相机的设备，装置或计算系统的图像信号处理器可以对从相机的图像传感器接收的图像数据应用分段透视变换。 图像信号处理器（ISP）的缩放单元可以对捕获的图像执行分段透视变换，以校正滚动快门伪影并提供视频图像稳定。 图像数据可以被划分为一系列水平切片，并且透视变换可以应用于每个切片。 变换可以基于以各种方式中的任何一种确定的运动数据，例如通过使用陀螺仪数据和/或光流计算。 分段透视变换可以被编码为数字差分分析器（DDA）步进器，并且可以使用可分离的标量运算来实现。 图像信号处理器可能不会将接收的图像数据写入系统存储器直到转换完成。

公开(公告)号：US20160037059A1

公开(公告)日：2016-02-04

申请号：US14449011

申请日：2014-07-31

Applicant: Apple Inc.

Inventor： Suk Hwan Lim ,  D. Amnon Silverstein

IPC: H04N5/232 ,  G06T5/20

CPC classification number: G06T5/20 ,  G06T5/002 ,  G06T5/50 ,  G06T2207/20024 ,  G06T2207/20182 ,  G06T2207/20201 ,  H04N5/217

Abstract: A temporal filter in an image processing pipeline may perform filtering using spatial filtering and noise history. A given pixel of a current image frame may be received for filtering at a temporal filter. A filtering weight may be determined for blending the given pixel with a corresponding pixel of a reference image frame that was previously filtered at the temporal filter. The filtering weight may be determined based on neighboring pixels of the given pixel in the current image frame and corresponding pixels in the reference image frame. The filtering weight may be adjusted according to a quality score indicating noise history for the corresponding pixel in the reference image frame. Based on the filtering weight, a filtered version of the given pixel may be generated, blending the given pixel and the corresponding pixel to store as part of a filtered version of the current image frame.

Abstract translation: 图像处理流水线中的时间滤波器可以使用空间滤波和噪声历史来执行滤波。 可以接收当前图像帧的给定像素以在时间滤波器处进行滤波。 可以确定用于将给定像素与先前在时间滤波器处被滤波的参考图像帧的对应像素混合的滤波权重。 可以基于当前图像帧中的给定像素和参考图像帧中的相应像素的相邻像素来确定滤波权重。 可以根据指示参考图像帧中的相应像素的噪声历史的质量得分来调整滤波权重。 基于过滤权重，可以生成给定像素的滤波版本，将给定像素和相应像素混合存储为当前图像帧的滤波版本的一部分。

公开(公告)号：US20160037058A1

公开(公告)日：2016-02-04

申请号：US14449010

申请日：2014-07-31

Applicant: Apple Inc.

Inventor： D. Amnon Silverstein ,  Suk Hwan Lim

IPC: H04N5/232 ,  G06T5/10

CPC classification number: H04N5/23229 ,  G06T1/20 ,  G06T5/002 ,  G06T5/10 ,  G06T5/50 ,  G06T2207/20024 ,  G06T2207/20182 ,  H04N5/21

Abstract: A temporal filter in an image processing pipeline may insert a frame delay when filtering an image frame. A given pixel of a current image frame may be received and a filtered version of the given pixel may be generated, blending the given pixel and a corresponding pixel of a reference image frame to store as part of a filtered version of the current image frame. If a frame delay setting is enabled, the corresponding pixel of the reference image frame may be provided as output for subsequent image processing inserting a frame delay for the current image frame. During the frame delay programming instructions may be received and image processing pipeline components may be configured according to the programming instructions. If the frame delay setting is disabled, then the filtered version of the given pixel may be provided as output for subsequent image processing.

Abstract translation: 图像处理流水线中的时间滤波器可以在对图像帧进行滤波时插入帧延迟。 可以接收当前图像帧的给定像素，并且可以生成给定像素的滤波版本，将给定像素和参考图像帧的对应像素混合以存储为当前图像帧的滤波版本的一部分。 如果启用了帧延迟设置，则可以将参考图像帧的相应像素提供为用于随后的图像处理的输出，为当前图像帧插入帧延迟。 在帧延迟期间，可以接收编程指令，并且可以根据编程指令来配置图像处理流水线部件。 如果帧延迟设置被禁用，则可以提供给定像素的滤波版本作为后续图像处理的输出。

公开(公告)号：US12198306B2

公开(公告)日：2025-01-14

申请号：US17950917

申请日：2022-09-22

Applicant: Apple Inc.

Inventor： Ching E. Ho ,  D. Amnon Silverstein

IPC: G06T5/73 ,  G06T3/4046 ,  G06T5/70 ,  G06T7/10

Abstract: Some embodiments relate to sharpening segments of an image differently based on content in the image. Content based sharpening is performed by a content image processing circuit that receives luminance values of an image and a content map. The content map identifies categories of content in segments of the image. Based on one or more of the identified categories of content, the circuit determines a content factor associated with a pixel. The content factor may also be based on a texture and/or chroma values. A texture value indicates a likelihood of a category of content and is based on detected edges in the image. A chroma value indicates a likelihood of a category of content and is based on color information of the image. The circuit receives the content factor and applies it to a version of the luminance value of the pixel to generate a sharpened version of the luminance value.

公开(公告)号：US11689826B2

公开(公告)日：2023-06-27

申请号：US17377308

申请日：2021-07-15

Applicant: Apple Inc.

Inventor： Guy Cote ,  D. Amnon Silverstein ,  Suk Hwan Lim ,  Sheng Lin ,  Haitao Guo

IPC: H04N5/217 ,  H04N25/67

CPC classification number: H04N25/67

Abstract: The present disclosure generally relates to systems and methods for image data processing. In certain embodiments, an image processing pipeline may be configured to receive a frame of the image data having a plurality of pixels acquired using a digital image sensor. The image processing pipeline may then be configured to determine a first plurality of correction factors that may correct each pixel in the plurality of pixels for fixed pattern noise. The first plurality of correction factors may be determined based at least in part on fixed pattern noise statistics that correspond to the frame of the image data. After determining the first plurality of correction factors, the image processing pipeline may be configured to configured to apply the first plurality of correction factors to the plurality of pixels, thereby reducing the fixed pattern noise present in the plurality of pixels.

公开(公告)号：US11330241B2

公开(公告)日：2022-05-10

申请号：US17227187

申请日：2021-04-09

Applicant: Apple Inc.

Inventor： D. Amnon Silverstein

IPC: H04N13/117 ,  G02B27/01 ,  G06F3/01

Abstract: Methods and apparatus for focusing in virtual reality (VR) or augmented reality (AR) devices based on gaze tracking information are described. Embodiments of a VR/AR head-mounted display (HMD) may include a gaze tracking system for detecting position and movement of the user's eyes. For AR applications, gaze tracking information may be used to direct external cameras to focus in the direction of the user's gaze so that the cameras focus on objects at which the user is looking. For AR or VR applications, the gaze tracking information may be used to adjust the focus of the eye lenses so that the virtual content that the user is currently looking at on the display has the proper vergence to match the convergence of the user's eyes.

公开(公告)号：US20210360176A1

公开(公告)日：2021-11-18

申请号：US17334409

申请日：2021-05-28

Applicant: Apple Inc.

Inventor： Sheng Lin ,  D. Amnon Silverstein

IPC: H04N5/347 ,  H04N5/367 ,  H04N5/232 ,  G06T5/00 ,  H04N9/64 ,  H04N9/04 ,  H04N5/345

Abstract: Embodiments relate to image signal processors (ISP) that include binner circuits that down-sample an input image. An input image may include a plurality of pixels. The output image of the binner circuit may include a reduced number of pixels. The binner circuit may include a plurality of different operation modes. In a bin mode, the binner circuit may blend a subset of input pixel values to generate an output pixel quad. In a skip mode, the binner circuit may select one of the input pixel values as the output pixel pixel. The selection may be performed randomly to avoid aliasing. In a luminance mode, the binner circuit may take a weighted average of a subset of pixel values having different colors. In a color value mode, the binner circuit may select one of the colors in a subset of pixel values as an output pixel value.