公开(公告)号：US20120114037A1

公开(公告)日：2012-05-10

申请号：US13348262

申请日：2012-01-11

申请人： Simon Winder ,  Matthew T. Uyttendaele ,  Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

发明人： Simon Winder ,  Matthew T. Uyttendaele ,  Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

IPC分类号： H04N11/02

CPC分类号： H04N21/4347 ,  H04N13/111 ,  H04N19/109 ,  H04N19/39 ,  H04N19/593 ,  H04N19/597 ,  H04N19/61 ,  H04N19/70 ,  H04N19/96 ,  H04N21/2365

摘要： A process for compressing and decompressing non-keyframes in sequential sets of contemporaneous video frames making up multiple video streams where the video frames in a set depict substantially the same scene from different viewpoints. Each set of contemporaneous video frames has a plurality frames designated as keyframes with the remaining being non-keyframes. In one embodiment, the non-keyframes are compressed using a multi-directional spatial prediction technique. In another embodiment, the non-keyframes of each set of contemporaneous video frames are compressed using a combined chaining and spatial prediction compression technique. The spatial prediction compression technique employed can be a single direction technique where just one reference frame, and so one chain, is used to predict each non-keyframe, or it can be a multi-directional technique where two or more reference frames, and so chains, are used to predict each non-keyframe.

摘要翻译： 一种用于在构成多个视频流的同步视频帧的顺序集合中压缩和解压缩非关键帧的过程，其中集合中的视频帧从不同视点描绘基本上相同的场景。 每组同时期的视频帧具有指定为关键帧的多个帧，其余的是非关键帧。 在一个实施例中，使用多方向空间预测技术来压缩非关键帧。 在另一个实施例中，使用组合链接和空间预测压缩技术来压缩每组同时期视频帧的非关键帧。 所使用的空间预测压缩技术可以是单向技术，其中仅使用一个参考帧，因此使用一条链来预测每个非关键帧，或者它可以是多方向技术，其中两个或更多个参考帧等 链，用于预测每个非关键帧。

公开(公告)号：US07015926B2

公开(公告)日：2006-03-21

申请号：US10879235

申请日：2004-06-28

申请人： Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang ,  Matthew T. Uyttendaele ,  Simon Winder

发明人： Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang ,  Matthew T. Uyttendaele ,  Simon Winder

IPC分类号： G09G5/02

CPC分类号： G06T15/205

摘要： A system and process for generating a two-layer, 3D representation of a digital or digitized image from the image and a pixel disparity map of the image is presented. The two layer representation includes a main layer having pixels exhibiting background colors and background disparities associated with correspondingly located pixels of depth discontinuity areas in the image, as well as pixels exhibiting colors and disparities associated with correspondingly located pixels of the image not found in these depth discontinuity areas. The other layer is a boundary layer made up of pixels exhibiting foreground colors, foreground disparities and alpha values associated with the correspondingly located pixels of the depth discontinuity areas. The depth discontinuity areas correspond to prescribed sized areas surrounding depth discontinuities found in the image using a disparity map thereof.

摘要翻译： 提出了一种用于从图像生成数字或数字化图像的二层3D表示和图像的像素视差图的系统和过程。 两层表示包括具有显示背景颜色的像素和与图像中的深度不连续区域的相应定位的像素相关联的背景差异的主层以及与在这些深度中未找到的图像的相应定位的像素相关联的颜色和差异的像素 不连续区域。 另一层是由与前述深度不连续区域的对应位置的像素相关联的前景色，前景差异和α值的像素构成的边界层。 深度不连续区域对应于使用其视差图在图像中发现的围绕深度不连续性的规定尺寸的区域。

公开(公告)号：US08121434B2

公开(公告)日：2012-02-21

申请号：US12138454

申请日：2008-06-13

申请人： Richard Szeliski ,  Matthew T. Uyttendaele ,  Simon Winder

发明人： Richard Szeliski ,  Matthew T. Uyttendaele ,  Simon Winder

IPC分类号： G06K9/32 ,  G06K9/40 ,  G09G5/00

CPC分类号： G06T3/4007 ,  G06T3/608

摘要： Multi-pass image resampling technique embodiments are presented that employ a series of one-dimensional filtering, resampling, and shearing stages to achieve good efficiency while maintaining high visual fidelity. In one embodiment, high-quality (multi-tap) image filtering is used inside each one-dimensional resampling stage. Because each stage only uses one-dimensional filtering, the overall computation efficiency is very good and amenable to graphics processing unit (GPU) implementation using pixel shaders. This embodiment also upsamples the image before shearing steps in a direction orthogonal to the shearing to prevent aliasing, and then downsamples the image to its final size with high-quality low-pass filtering. This ensures that none of the stages causes excessive blurring or aliasing.

摘要翻译： 提出了采用一系列一维滤波，重采样和剪切阶段以实现高效率同时保持高视觉保真度的多通图像重采样技术实施例。 在一个实施例中，在每个一维重采样阶段内使用高质量（多抽头）图像滤波。 因为每个阶段只使用一维过滤，所以整体计算效率非常好，并且适合使用像素着色器的图形处理单元（GPU）实现。 该实施例还在与剪切正交的方向上的剪切步骤之前对图像进行上采样以防止混叠，然后通过高质量低通滤波将图像下采样到其最终尺寸。 这样可以确保任何一个阶段都不会引起过度的模糊或混叠。

公开(公告)号：US07702016B2

公开(公告)日：2010-04-20

申请号：US11097533

申请日：2005-03-31

申请人： Simon Winder ,  Matthew T. Uyttendaele ,  Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

发明人： Simon Winder ,  Matthew T. Uyttendaele ,  Charles Lawrence Zitnick, III ,  Richard Szeliski ,  Sing Bing Kang

IPC分类号： H04B1/66 ,  H04N7/12 ,  H04N11/02 ,  H04N11/04 ,  H04N5/14 ,  H04N9/64

CPC分类号： H04N21/4347 ,  H04N13/111 ,  H04N19/109 ,  H04N19/39 ,  H04N19/593 ,  H04N19/597 ,  H04N19/61 ,  H04N19/70 ,  H04N19/96 ,  H04N21/2365

摘要： A system and process for compressing and decompressing multiple video streams depicting substantially the same dynamic scene from different viewpoints that from a grid of viewpoints. Each frame in each contemporaneous set of video frames of the multiple streams is represented by at least a two layers—a main layer and a boundary layer. Compression of the main layers involves first designating one or more of these layers in each set of contemporaneous frames as keyframes. For each set of contemporaneous frames in time sequence order, the main layer of each keyframe is compressed using an inter-frame compression technique. In addition, the main layer of each non-keyframe within the frame set under consideration is compressed using a spatial prediction compression technique. Finally, the boundary layers of each frame in the current frame set are each compressed using an intra-frame compression technique. Decompression is generally the reverse of the compression process.

摘要翻译： 一种用于压缩和解压缩从视点网格的不同观点描绘基本相同的动态场景的多个视频流的系统和过程。 多个流的每个同期视频帧集合中的每个帧由至少两层（主层和边界层）表示。 主要层的压缩包括首先将每组同期帧中的这些层中的一个或多个指定为关键帧。 对于按时间顺序排列的每组同期帧，使用帧间压缩技术对每个关键帧的主层进行压缩。 另外，使用空间预测压缩技术对所考虑的帧集合内的每个非关键帧的主层进行压缩。 最后，使用帧内压缩技术对当前帧集合中每帧的边界层进行压缩。 压缩通常与压缩过程相反。

公开(公告)号：US20090310888A1

公开(公告)日：2009-12-17

申请号：US12138454

申请日：2008-06-13

申请人： Richard Szeliski ,  Matthew T. Uyttendaele ,  Simon Winder

发明人： Richard Szeliski ,  Matthew T. Uyttendaele ,  Simon Winder

IPC分类号： G06K9/32

CPC分类号： G06T3/4007 ,  G06T3/608

摘要： Multi-pass image resampling technique embodiments are presented that employ a series of one-dimensional filtering, resampling, and shearing stages to achieve good efficiency while maintaining high visual fidelity. In one embodiment, high-quality (multi-tap) image filtering is used inside each one-dimensional resampling stage. Because each stage only uses one-dimensional filtering, the overall computation efficiency is very good and amenable to graphics processing unit (GPU) implementation using pixel shaders. This embodiment also upsamples the image before shearing steps in a direction orthogonal to the shearing to prevent aliasing, and then downsamples the image to its final size with high-quality low-pass filtering. This ensures that none of the stages causes excessive blurring or aliasing.

摘要翻译： 提出了采用一系列一维滤波，重采样和剪切阶段以实现高效率同时保持高视觉保真度的多通图像重采样技术实施例。 在一个实施例中，在每个一维重采样阶段内使用高质量（多抽头）图像滤波。 因为每个阶段只使用一维过滤，所以整体计算效率非常好，并且适合使用像素着色器的图形处理单元（GPU）实现。 该实施例还在与剪切正交的方向上的剪切步骤之前对图像进行上采样以防止混叠，然后通过高质量低通滤波将图像下采样到其最终尺寸。 这样可以确保任何一个阶段都不会引起过度的模糊或混叠。

公开(公告)号：US06879731B2

公开(公告)日：2005-04-12

申请号：US10425338

申请日：2003-04-29

申请人： Sing Bing Kang ,  Matthew T. Uyttendaele ,  Simon Winder ,  Richard Szeliski

发明人： Sing Bing Kang ,  Matthew T. Uyttendaele ,  Simon Winder ,  Richard Szeliski

IPC分类号： H04N5/235 ,  H04N5/77 ,  H04N5/781 ,  H04N5/85 ,  G06K9/40

CPC分类号： H04N5/2355 ,  G06T5/50 ,  H04N5/235 ,  H04N5/2352 ,  H04N5/77 ,  H04N5/781 ,  H04N5/85

摘要： A system and process for generating High Dynamic Range (HDR) video is presented which involves first capturing a video image sequence while varying the exposure so as to alternate between frames having a shorter and longer exposure. The exposure for each frame is set prior to it being captured as a function of the pixel brightness distribution in preceding frames. Next, for each frame of the video, the corresponding pixels between the frame under consideration and both preceding and subsequent frames are identified. For each corresponding pixel set, at least one pixel is identified as representing a trustworthy pixel. The pixel color information associated with the trustworthy pixels is then employed to compute a radiance value for each pixel set to form a radiance map. A tone mapping procedure can then be performed to convert the radiance map into an 8-bit representation of the HDR frame.

摘要翻译： 提出了用于产生高动态范围（HDR）视频的系统和过程，其涉及首先在改变曝光的同时捕获视频图像序列，以便在具有较短和较长曝光的帧之间交替。 每个帧的曝光在其被捕获之前被设置为在先前帧中的像素亮度分布的函数。 接下来，对于视频的每个帧，识别所考虑的帧与前后帧之间的对应像素。 对于每个对应的像素集合，至少一个像素被识别为表示可靠的像素。 然后使用与可信赖像素相关联的像素颜色信息来计算每个像素组的辐射值以形成辐射图。 然后可以执行色调映射过程以将辐射图转换成HDR帧的8位表示。

公开(公告)号：US20130095920A1

公开(公告)日：2013-04-18

申请号：US13273213

申请日：2011-10-13

申请人： Kestutis Patiejunas ,  Kanchan Mitra ,  Patrick Sweeney ,  Yaron Eshet ,  Adam G. Kirk ,  Sing Bing Kang ,  Charles Lawrence Zitnick, III ,  David Eraker ,  David Harnett ,  Amit Mital ,  Simon Winder

发明人： Kestutis Patiejunas ,  Kanchan Mitra ,  Patrick Sweeney ,  Yaron Eshet ,  Adam G. Kirk ,  Sing Bing Kang ,  Charles Lawrence Zitnick, III ,  David Eraker ,  David Harnett ,  Amit Mital ,  Simon Winder

IPC分类号： A63F13/00 ,  H04N13/02

CPC分类号： G06T15/00 ,  G06T7/521 ,  G06T7/593 ,  G06T15/04 ,  G06T17/20 ,  G06T2207/10021 ,  G06T2207/10024 ,  G06T2207/10048 ,  G06T2207/20228 ,  H04N13/111 ,  H04N13/271 ,  H04N2013/0081

摘要： Methods and systems for generating free viewpoint video using an active infrared (IR) stereo module are provided. The method includes computing a depth map for a scene using an active IR stereo module. The depth map may be computed by projecting an IR dot pattern onto the scene, capturing stereo images from each of two or more synchronized IR cameras, detecting dots within the stereo images, computing feature descriptors corresponding to the dots in the stereo images, computing a disparity map between the stereo images, and generating the depth map using the disparity map. The method also includes generating a point cloud for the scene using the depth map, generating a mesh of the point cloud, and generating a projective texture map for the scene from the mesh of the point cloud. The method further includes generating the video for the scene using the projective texture map.

摘要翻译： 提供了使用主动红外（IR）立体声模块产生免费视点视频的方法和系统。 该方法包括使用主动IR立体声模块来计算场景的深度图。 可以通过将IR点图案投影到场景上来计算深度图，从两个或更多个同步红外相机中的每一个拍摄立体图像，检测立体图像内的点，计算与立体图像中的点相对应的特征描述符， 立体图像之间的视差图，并使用视差图生成深度图。 该方法还包括使用深度图生成场景的点云，生成点云的网格，并从点云的网格生成场景的投影纹理贴图。 该方法还包括使用投影纹理图生成场景的视频。

公开(公告)号：US08126275B2

公开(公告)日：2012-02-28

申请号：US11739672

申请日：2007-04-24

申请人： Simon Winder

发明人： Simon Winder

IPC分类号： G06K9/46

CPC分类号： G06K9/4671

摘要： An interest point detection technique is presented. More particularly, for each of possibly multiple image pyramid resolutions, a cornerness image is generated. One or more potential interest point locations are identified in the cornerness image. This involves finding locations associated with a pixel that exhibits a higher corner strength value than pixels in a prescribed-sized surrounding pixel neighborhood. The potential interest point locations are then clustered to identify groups that likely derive from a same 2D structure. Potential interest point locations in one or more of the identified groups are respectively combined to produce a single location that represents the combined group. The representative location of each group having one is then designated as an interest point. An optional location refinement can also be implemented.

摘要翻译： 提出了一种兴趣点检测技术。 更具体地，对于可能的多个图像金字塔分辨率中的每一个，生成角部图像。 在角度图像中识别一个或多个潜在的兴趣点位置。 这涉及找到与规定大小的周围像素邻域中的像素相比具有更高角强度值的像素的位置。 然后将潜在的兴趣点位置聚类以识别可能来自相同2D结构的组。 分别组合一个或多个识别组中的潜在兴趣点位置以产生表示组合组的单个位置。 然后，将具有一个的每个组的代表性位置指定为兴趣点。 还可以实现可选的位置细化。

公开(公告)号：US20130321586A1

公开(公告)日：2013-12-05

申请号：US13588917

申请日：2012-08-17

申请人： Adam Kirk ,  Patrick Sweeney ,  Don Gillett ,  Neil Fishman ,  Kanchan Mitra ,  Amit Mital ,  David Harnett ,  Yaron Eshet ,  Simon Winder ,  David Eraker

发明人： Adam Kirk ,  Patrick Sweeney ,  Don Gillett ,  Neil Fishman ,  Kanchan Mitra ,  Amit Mital ,  David Harnett ,  Yaron Eshet ,  Simon Winder ,  David Eraker

IPC分类号： H04N13/02

CPC分类号： G06T15/04 ,  G06T15/08 ,  G06T15/205 ,  G06T17/00 ,  G06T2210/56 ,  H04N7/142 ,  H04N7/15 ,  H04N7/157 ,  H04N13/117 ,  H04N13/194 ,  H04N13/239 ,  H04N13/243 ,  H04N13/246 ,  H04N13/257 ,  H04R2227/005 ,  H04S2400/15

摘要： Cloud based FVV streaming technique embodiments presented herein generally employ a cloud based FVV pipeline to create, render and transmit FVV frames depicting a captured scene as would be viewed from a current synthetic viewpoint selected by an end user and received from a client computing device. The FVV frames use a similar level of bandwidth as a conventional streaming movie would consume. To change viewpoints, a new viewpoint is sent from the client to the cloud, and a new streaming movie is initiated from the new viewpoint. Frames associated with that viewpoint are created, rendered and transmitted to the client until a new viewpoint request is received.

摘要翻译： 本文呈现的基于云的FVV流技术实施例通常使用基于云的FVV流水线来创建，呈现和发送描绘所捕获场景的FVV帧，如将从最终用户选择并从客户端计算设备接收的当前合成视点所观察到的。 FVV帧使用与传统流式电影将消耗的类似的带宽水平。 为了改变观点，从客户端向云端发送一个新的观点，从新的角度出发新的流媒体电影。 与该视点相关联的帧被创建，呈现并发送到客户端，直到接收到新的视点请求为止。

公开(公告)号：US20130321396A1

公开(公告)日：2013-12-05

申请号：US13599170

申请日：2012-08-30

申请人： Adam Kirk ,  Kanchan Mitra ,  Patrick Sweeney ,  Don Gillett ,  Neil Fishman ,  Simon Winder ,  Yaron Eshet ,  David Harnett ,  Amit Mital ,  David Eraker

发明人： Adam Kirk ,  Kanchan Mitra ,  Patrick Sweeney ,  Don Gillett ,  Neil Fishman ,  Simon Winder ,  Yaron Eshet ,  David Harnett ,  Amit Mital ,  David Eraker

IPC分类号： G06T15/00

CPC分类号： G06T15/04 ,  G06T15/08 ,  G06T15/205 ,  G06T17/00 ,  G06T2210/56 ,  H04N7/142 ,  H04N7/15 ,  H04N7/157 ,  H04N13/117 ,  H04N13/194 ,  H04N13/239 ,  H04N13/243 ,  H04N13/246 ,  H04N13/257 ,  H04R2227/005 ,  H04S2400/15

摘要： Free viewpoint video of a scene is generated and presented to a user. An arrangement of sensors generates streams of sensor data each of which represents the scene from a different geometric perspective. The sensor data streams are calibrated. A scene proxy is generated from the calibrated sensor data streams. The scene proxy geometrically describes the scene as a function of time and includes one or more types of geometric proxy data which is matched to a first set of current pipeline conditions in order to maximize the photo-realism of the free viewpoint video resulting from the scene proxy at each point in time. A current synthetic viewpoint of the scene is generated from the scene proxy. This viewpoint generation maximizes the photo-realism of the current synthetic viewpoint based upon a second set of current pipeline conditions. The current synthetic viewpoint is displayed.

摘要翻译： 生成场景的自由视点视频并呈现给用户。 传感器的布置产生传感器数据流，每个传感器数据从不同的几何视角表示场景。 传感器数据流被校准。 从校准的传感器数据流生成场景代理。 场景代理将场景几何描述为时间的函数，并且包括与当前流水线条件的第一组匹配的一种或多种类型的几何代理数据，以便最大化场景产生的自由视点视频的照片真实性 代理在每个时间点。 从场景代理产生现场的合成视点。 该视点生成基于第二组当前流水线条件使当前合成视点的照片真实性最大化。 显示当前的综合观点。