公开(公告)号：US07206000B2

公开(公告)日：2007-04-17

申请号：US11334591

申请日：2006-01-17

Applicant: Charles Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang ,  Matthew Uyttendaele ,  Simon Winder

Inventor： Charles Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang ,  Matthew Uyttendaele ,  Simon Winder

IPC: G09G5/02

CPC classification number: G06T15/205

Abstract: A system and process for generating a two-layer, 3D representation of a digital or digitized image from the image and a pixel disparity map of the image is presented. The two layer representation includes a main layer having pixels exhibiting background colors and background disparities associated with correspondingly located pixels of depth discontinuity areas in the image, as well as pixels exhibiting colors and disparities associated with correspondingly located pixels of the image not found in these depth discontinuity areas. The other layer is a boundary layer made up of pixels exhibiting foreground colors, foreground disparities and alpha values associated with the correspondingly located pixels of the depth discontinuity areas. The depth discontinuity areas correspond to prescribed sized areas surrounding depth discontinuities found in the image using a disparity map thereof.

Abstract translation: 提出了一种用于从图像生成数字或数字化图像的二层3D表示和图像的像素视差图的系统和过程。 两层表示包括具有显示背景颜色的像素和与图像中的深度不连续区域的相应定位的像素相关联的背景差异的主层以及与在这些深度中未找到的图像的相应定位的像素相关联的颜色和差异的像素 不连续区域。 另一层是由与前述深度不连续区域的对应位置的像素相关联的前景色，前景差异和α值的像素构成的边界层。 深度不连续区域对应于使用其视差图在图像中发现的围绕深度不连续性的规定尺寸的区域。

公开(公告)号：US07124066B2

公开(公告)日：2006-10-17

申请号：US10837424

申请日：2004-04-30

Applicant: Stephen Marschner ,  Brian K. Guenter ,  Sashi Raghupathy ,  Kirk Olynyk ,  Sing Bing Kang

Inventor： Stephen Marschner ,  Brian K. Guenter ,  Sashi Raghupathy ,  Kirk Olynyk ,  Sing Bing Kang

IPC: G06F17/10 ,  G06K9/00

CPC classification number: G06K9/4661 ,  G06K9/00268 ,  G06T7/521 ,  G06T15/506

Abstract: In the described embodiment, methods and systems for processing facial image data for use in animation are described. In one embodiment, a system is provided that illuminates a face with illumination that is sufficient to enable the simultaneous capture of both structure data, e.g. a range or depth map, and reflectance properties, e.g. the diffuse reflectance of a subject's face. This captured information can then be used for various facial animation operations, among which are included expression recognition and expression transformation.

公开(公告)号：US06985156B2

公开(公告)日：2006-01-10

申请号：US10875741

申请日：2004-06-23

Applicant: Lifeng Wang ,  Sing Bing Kang ,  Richard Szeliski ,  Heung-Yeung Shum ,  Baining Guo

Inventor： Lifeng Wang ,  Sing Bing Kang ,  Richard Szeliski ,  Heung-Yeung Shum ,  Baining Guo

IPC: G09G5/00

CPC classification number: G06T11/001

Abstract: A system and process for reconstructing optimal texture maps from multiple views of a scene is described. In essence, this reconstruction is based on the optimal synthesis of textures from multiple sources. This is generally accomplished using basic image processing theory to derive the correct weights for blending the multiple views. Namely, the steps of reconstructing, warping, prefiltering, and resampling are followed in order to warp reference textures to a desired location, and to compute spatially-variant weights for optimal blending. These weights take into consideration the anisotropy in the texture projection and changes in sampling frequency due to foreshortening. The weights are combined and the computation of the optimal texture is treated as a restoration problem, which involves solving a linear system of equations. This approach can be incorporated in a variety of applications, such as texturing of 3D models, analysis by synthesis methods, super-resolution techniques, and view-dependent texture mapping.

Abstract translation: 描述用于从场景的多个视图重建最佳纹理图的系统和过程。 实质上，这种重建是基于来自多个源的纹理的最佳合成。 这通常使用基本图像处理理论来实现，以导出用于混合多个视图的正确权重。 即，遵循重构，翘曲，预过滤和重采样的步骤，以便将参考纹理扭曲到期望的位置，并计算用于最佳混合的空间变体权重。 这些权重考虑到纹理投影中的各向异性和由于缩短引起的采样频率的变化。 权重相结合，最优纹理的计算被视为恢复问题，其涉及求解线性方程组。 这种方法可以并入各种应用中，例如3D模型的纹理化，通过合成方法的分析，超分辨率技术和视图相关的纹理映射。

公开(公告)号：US09414048B2

公开(公告)日：2016-08-09

申请号：US13315488

申请日：2011-12-09

Applicant: Kevin Robert Karsch ,  Ce Liu ,  Sing Bing Kang

Inventor： Kevin Robert Karsch ,  Ce Liu ,  Sing Bing Kang

IPC: H04N15/00 ,  H04N13/00 ,  H04N13/02

CPC classification number: H04N13/261

Abstract: In general, a “Stereoscopic Video Converter” (SVC) provides various techniques for automatically converting arbitrary 2D video sequences into perceptually plausible stereoscopic or “3D” versions while optionally generating dense depth maps for every frame of the video sequence. In particular, the automated 2D-to-3D conversion process first automatically estimates scene depth for each frame of an input video sequence via a label transfer process that matches features extracted from those frames with features from a database of images and videos having known ground truth depths. The estimated depth distributions for all image frames of the input video sequence are then used by the SVC for automatically generating a “right view” of a corresponding stereoscopic image for each frame (assuming that each original input frame represents the “left view” of the stereoscopic image).

Abstract translation: 通常，“立体视频转换器”（SVC）提供了用于将任意2D视频序列自动转换为听觉上合理的立体或“3D”版本的各种技术，同时可选地为视频序列的每一帧产生密集的深度图。 特别地，自动2D到3D转换过程首先通过标签传送过程自动地估计输入视频序列的每帧的场景深度，所述标签传送过程与从具有已知地面实况的图像和视频的数据库的特征相匹配的特征提取出匹配的特征 深度。 输入视频序列的所有图像帧的估计深度分布然后由SVC用于自动生成每帧的对应立体图像的“右视图”（假设每个原始输入帧表示“ 立体图像）。

公开(公告)号：US09275680B2

公开(公告)日：2016-03-01

申请号：US12485179

申请日：2009-06-16

Applicant: Charles Lawrence Zitnick, III ,  Bryan K. Ressler ,  Sing Bing Kang ,  Michael F. Cohen ,  Jagannatha Koppal

Inventor： Charles Lawrence Zitnick, III ,  Bryan K. Ressler ,  Sing Bing Kang ,  Michael F. Cohen ,  Jagannatha Koppal

IPC: G06F3/048 ,  G11B27/034 ,  G11B27/34 ,  H04N21/485 ,  H04N21/6377 ,  H04N21/658

CPC classification number: G11B27/034 ,  G11B27/34 ,  H04N21/4854 ,  H04N21/6377 ,  H04N21/658

Abstract: Described is a user interface that displays a representation of a stereo scene, and includes interactive mechanisms for changing parameter values that determine the perceived appearance of that scene. The scene is modeled as if viewed from above, including a representation of a viewer's eyes, a representation of a viewing screen, and an indication simulating what each of the viewer eyes perceives on the viewing screen. Variable parameters may include a vergence parameter, a dolly parameter, a field-of-view parameter, an interocular parameter and a proscenium arch parameter.

Abstract translation: 描述了显示立体场景的表示的用户界面，并且包括用于改变确定该场景的感知外观的参数值的交互机制。 该场景被建模为仿佛从上方观看，包括观看者的眼睛的表示，观看屏幕的表示，以及模拟观看者眼睛在观看屏幕上感知的每一个的指示。 可变参数可以包括聚集参数，小车参数，视野参数，眼镜参数和前景拱参数。

公开(公告)号：US20130095920A1

公开(公告)日：2013-04-18

申请号：US13273213

申请日：2011-10-13

Applicant: Kestutis Patiejunas ,  Kanchan Mitra ,  Patrick Sweeney ,  Yaron Eshet ,  Adam G. Kirk ,  Sing Bing Kang ,  Charles Lawrence Zitnick, III ,  David Eraker ,  David Harnett ,  Amit Mital ,  Simon Winder

Inventor： Kestutis Patiejunas ,  Kanchan Mitra ,  Patrick Sweeney ,  Yaron Eshet ,  Adam G. Kirk ,  Sing Bing Kang ,  Charles Lawrence Zitnick, III ,  David Eraker ,  David Harnett ,  Amit Mital ,  Simon Winder

IPC: A63F13/00 ,  H04N13/02

CPC classification number: G06T15/00 ,  G06T7/521 ,  G06T7/593 ,  G06T15/04 ,  G06T17/20 ,  G06T2207/10021 ,  G06T2207/10024 ,  G06T2207/10048 ,  G06T2207/20228 ,  H04N13/111 ,  H04N13/271 ,  H04N2013/0081

Abstract: Methods and systems for generating free viewpoint video using an active infrared (IR) stereo module are provided. The method includes computing a depth map for a scene using an active IR stereo module. The depth map may be computed by projecting an IR dot pattern onto the scene, capturing stereo images from each of two or more synchronized IR cameras, detecting dots within the stereo images, computing feature descriptors corresponding to the dots in the stereo images, computing a disparity map between the stereo images, and generating the depth map using the disparity map. The method also includes generating a point cloud for the scene using the depth map, generating a mesh of the point cloud, and generating a projective texture map for the scene from the mesh of the point cloud. The method further includes generating the video for the scene using the projective texture map.

Abstract translation: 提供了使用主动红外（IR）立体声模块产生免费视点视频的方法和系统。 该方法包括使用主动IR立体声模块来计算场景的深度图。 可以通过将IR点图案投影到场景上来计算深度图，从两个或更多个同步红外相机中的每一个拍摄立体图像，检测立体图像内的点，计算与立体图像中的点相对应的特征描述符， 立体图像之间的视差图，并使用视差图生成深度图。 该方法还包括使用深度图生成场景的点云，生成点云的网格，并从点云的网格生成场景的投影纹理贴图。 该方法还包括使用投影纹理图生成场景的视频。

公开(公告)号：US07884823B2

公开(公告)日：2011-02-08

申请号：US11761604

申请日：2007-06-12

Applicant: Joe Bertolami ,  Robert M. Craig ,  Dax Hawkins ,  Sing Bing Kang ,  Jonathan E. Lange

Inventor： Joe Bertolami ,  Robert M. Craig ,  Dax Hawkins ,  Sing Bing Kang ,  Jonathan E. Lange

IPC: G06T15/20 ,  G06T15/00

CPC classification number: A63F13/525 ,  A63F13/10 ,  A63F2300/66 ,  G06T15/20 ,  H04N13/275

Abstract: Game data is rendered in three dimensions in the GPU of a game console. A left camera view and a right camera view are generated from a single camera view. The left and right camera positions are derived as an offset from a default camera. The focal distance of the left and right cameras is infinity. A game developer does not have to encode dual images into a specific hardware format. When a viewer sees the two slightly offset images, the user's brain combines the two offset images into a single 3D image to give the illusion that objects either pop out from or recede into the display screen. In another embodiment, individual, private video is rendered, on a single display screen, for different viewers. Rather than rendering two similar offset images, two completely different images are rendered allowing each player to view only one of the images.

Abstract translation: 游戏数据在游戏机的GPU中呈现三维。 从单个摄像机视图生成左侧摄像机视图和右侧摄像机视图。 左和右摄像机位置被派生为与默认摄像机的偏移量。 左右相机的焦距为无穷远。 游戏开发者不必将双重图像编码为特定的硬件格式。 当观众看到两个轻微偏移的图像时，用户的大脑将两个偏移图像组合成单个3D图像，以给出对象从显示屏幕中弹出或退回到显示屏幕的错觉。 在另一个实施例中，单独的专用视频在单个显示屏幕上被呈现给不同的观看者。 而不是渲染两个相似的偏移图像，渲染两个完全不同的图像，允许每个播放器仅查看其中一个图像。

公开(公告)号：US07808532B2

公开(公告)日：2010-10-05

申请号：US11807448

申请日：2007-05-29

Applicant: Jian Sun ,  Jian Sun ,  Sing Bing Kang ,  Xiaoou Tang ,  Heung-Yeung Shum

Inventor： Jian Sun ,  Jian Sun ,  Sing Bing Kang ,  Xiaoou Tang ,  Heung-Yeung Shum

IPC: H04N9/73

CPC classification number: H04N9/76 ,  G06T7/11 ,  G06T7/143 ,  G06T7/194 ,  G06T2207/10144 ,  H04N5/23232

Abstract: A flash-based strategy is used to separate foreground information from background information within image information. In this strategy, a first image is taken without the use of flash. A second image is taken of the same subject matter with the use of flash. The foreground information in the flash image is illuminated by the flash to a much greater extent than the background information. Based on this property, the strategy applies processing to extract the foreground information from the background information. The strategy supplements the flash information by also taking into consideration motion information and color information.

Abstract translation: 基于闪存的策略用于将前景信息与图像信息中的背景信息分离。 在这个策略中，第一个图像是不使用闪光灯的。 使用闪光灯拍摄相同主题的第二张照片。 闪光灯中的前景信息被闪光灯照亮到比背景信息更大的程度。 基于此属性，该策略应用处理从背景信息中提取前景信息。 该策略通过考虑运动信息和颜色信息来补充闪光信息。

公开(公告)号：US20100111417A1

公开(公告)日：2010-05-06

申请号：US12263618

申请日：2008-11-03

Applicant: Benjamin Ward ,  Sing Bing Kang ,  Eric Bennett

Inventor： Benjamin Ward ,  Sing Bing Kang ,  Eric Bennett

IPC: G06K9/34

CPC classification number: G06T7/579 ,  G06T2207/10012 ,  G06T2207/10016 ,  G06T2207/20104 ,  H04N13/261

Abstract: Two-dimensional (2D) video is converted into multi-view video. The 2D video is segmented to generate a temporally consistent segmented 2D video which is made up of a sequence of segmented frames. The multi-view video is generated by employing user-guided operations to generate depth assignments for the segments associated with user-assigned regions of the segmented frames, where a user-assigned region is formed from a group of contiguous segments selected by the user.

Abstract translation: 二维（2D）视频转换为多视角视频。 2D视频被分割以产生由分段帧序列组成的时间上一致的分割的2D视频。 多视点视频是通过采用用户指导的操作来生成与分段帧的用户分配区域相关联的片段的深度分配，其中由用户选择的一组连续片段形成用户分配的区域。

公开(公告)号：US07561620B2

公开(公告)日：2009-07-14

申请号：US10910077

申请日：2004-08-03

Applicant: Simon Winder ,  Matthew Uyttendaele ,  Charles Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

Inventor： Simon Winder ,  Matthew Uyttendaele ,  Charles Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

IPC: H04B1/66 ,  H04N7/12 ,  H04N11/02 ,  H04N11/04 ,  H04N5/14 ,  H04N9/64

CPC classification number: H04N21/4347 ,  H04N13/111 ,  H04N19/109 ,  H04N19/39 ,  H04N19/593 ,  H04N19/597 ,  H04N19/61 ,  H04N19/70 ,  H04N19/96 ,  H04N21/2365

Abstract: A system and process for compressing and decompressing multiple video streams depicting substantially the same dynamic scene from different viewpoints. Each frame in each contemporaneous set of video frames of the multiple streams is represented by at least a two layers—a main layer and a boundary layer. Compression of the main layers involves first designating one or more of these layers in each set of contemporaneous frames as keyframes. For each set of contemporaneous frames in time sequence order, the main layer of each keyframe is compressed using an inter-frame compression technique. In addition, the main layer of each non-keyframe within the frame set under consideration is compressed using a spatial prediction compression technique. Finally, the boundary layers of each frame in the current frame set are each compressed using an intra-frame compression technique. Decompression is generally the reverse of the compression process.

Abstract translation: 一种用于压缩和解压缩从不同观点描绘基本相同的动态场景的多个视频流的系统和过程。 多个流的每个同期视频帧集合中的每个帧由至少两层（主层和边界层）表示。 主要层的压缩包括首先将每组同期帧中的这些层中的一个或多个指定为关键帧。 对于按时间顺序排列的每组同期帧，使用帧间压缩技术对每个关键帧的主层进行压缩。 另外，使用空间预测压缩技术对所考虑的帧集合内的每个非关键帧的主层进行压缩。 最后，使用帧内压缩技术对当前帧集合中每帧的边界层进行压缩。 压缩通常与压缩过程相反。