公开(公告)号：US10552947B2

公开(公告)日：2020-02-04

申请号：US15824574

申请日：2017-11-28

Applicant: Google LLC

Inventor： Chia-Kai Liang ,  Kent Oberheu ,  Kurt Akeley ,  Garrett Girod ,  Nikhil Karnad ,  Francis A. Benevides

IPC: G06K9/00 ,  G06T5/00 ,  G06F3/0484 ,  G06T7/194 ,  G06T7/11

Abstract: An image such as a light-field image may be processed to provide depth-based blurring. The image may be received in a data store. At an input device, first and second user input may be received to designate a first focus depth and a second focus depth different from the first focus depth, respectively. A processor may identify one or more foreground portions of the image that have one or more foreground portion depths, each of which is less than the first focus depth. The processor may also identify one or more background portions of the image that have one or more background portion depths, each of which is greater than the second focus depth. The processor may also apply blurring to the one or more foreground portions and the one or more background portions to generate a processed image, which may be displayed on a display device.

公开(公告)号：US10341632B2

公开(公告)日：2019-07-02

申请号：US15590877

申请日：2017-05-09

Applicant: Google LLC.

Inventor： Derek Pang ,  Colvin Pitts ,  Kurt Akeley

IPC: G06F3/00 ,  G06F3/01 ,  G06F3/03 ,  H04N5/222 ,  H04N5/225 ,  H04N5/247 ,  H04N13/117 ,  H04N13/156 ,  H04N13/161 ,  H04N13/194 ,  H04N13/232 ,  H04N13/243 ,  H04N13/275 ,  H04N13/282 ,  H04N13/344

Abstract: An environment may be displayed from a viewpoint. According to one method, volumetric video data may be acquired depicting the environment, for example, using a tiled camera array. A plurality of vantages may be distributed throughout a viewing volume from which the environment is to be viewed. The volumetric video data may be used to generate video data for each vantage, representing the view of the environment from that vantage. User input may be received designating a viewpoint within the viewing volume. From among the plurality of vantages, a subset nearest to the viewpoint may be identified. The video data from the subset may be retrieved and combined to generate viewpoint video data depicting the environment from the viewpoint. The viewpoint video data may be displayed for the viewer to display a view of the environment from the viewpoint selected by the user.

公开(公告)号：US10546424B2

公开(公告)日：2020-01-28

申请号：US15729918

申请日：2017-10-11

Applicant: GOOGLE LLC

Inventor： Derek Pang ,  Colvin Pitts ,  Kurt Akeley

IPC: G06T19/00 ,  G06F3/01 ,  H04N13/117 ,  H04N13/139

Abstract: A virtual reality or augmented reality experience of a scene may be presented to a viewer using layered data retrieval and/or processing. A first layer of a video stream may be retrieved, and a first viewer position and/or orientation may be received. The first layer may be processed to generate first viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The first viewpoint video may be displayed for the viewer. Then, a second layer of the video stream may be retrieved, and a second viewer position and/or orientation may be received. The second layer may be processed to generate second viewpoint video of the scene from a second virtual viewpoint corresponding to the second viewer position and/or orientation, with higher quality than the first viewpoint video. The second viewpoint video may be displayed for the viewer.

公开(公告)号：US10419737B2

公开(公告)日：2019-09-17

申请号：US15706343

申请日：2017-09-15

Applicant: GOOGLE LLC

Inventor： Derek Pang ,  Colvin Pitts ,  Kurt Akeley

IPC: H04N13/117 ,  H04N13/156 ,  H04N13/161 ,  H04N13/243 ,  H04N13/282 ,  H04N13/344 ,  H04N5/222 ,  H04N5/225 ,  H04N5/247 ,  H04N19/597 ,  H04N13/232 ,  H04N19/46 ,  H04N19/42 ,  H04N19/33 ,  H04N13/275 ,  H04N19/147 ,  H04N19/436 ,  H04N19/895

Abstract: A video stream for a scene for a virtual reality or augmented reality experience may be stored and delivered to a viewer. The video stream may be divided into a plurality of units based on time segmentation, viewpoint segmentation, and/or view orientation segmentation. Each of the units may be divided into a plurality of sub-units based on a different segmentation from the units, via time segmentation, viewpoint segmentation, and/or view orientation segmentation. At least a portion of the video stream may be stored in a file that includes a plurality of the units. Each unit may be a group of pictures that is a sequence of successive frames in time. Each sub-unit may be a vantage defining a viewpoint from which the scene is viewable. Each vantage may be further divided into tiles, each of which is part of the vantage, limited to one or more particular view orientations.

公开(公告)号：US10897608B2

公开(公告)日：2021-01-19

申请号：US16032261

申请日：2018-07-11

Applicant: Google LLC

Inventor： Colvin Pitts ,  Chia-Kai Liang ,  Kurt Akeley

IPC: H04N13/229 ,  G02B3/00 ,  H04N13/218 ,  H04N13/232

Abstract: A light-field camera may generate four-dimensional light-field data indicative of incoming light. The light-field camera may have an aperture configured to receive the incoming light, an image sensor, and a microlens array configured to redirect the incoming light at the image sensor. The image sensor may receive the incoming light and, based on the incoming light, generate the four-dimensional light-field data, which may have first and second spatial dimensions and first and second angular dimensions. The first angular dimension may have a first resolution higher than a second resolution of the second angular dimension.

公开(公告)号：US10444931B2

公开(公告)日：2019-10-15

申请号：US15590841

申请日：2017-05-09

Applicant: Google LLC

Inventor： Kurt Akeley

IPC: G06T19/00 ,  G06F3/0481 ,  G06T13/40 ,  G06F3/01

Abstract: Video data of an environment may be prepared for presentation to a user in a virtual reality or augmented reality experience. According to one method, a plurality of locations distributed throughout a viewing volume may be designated, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations. For each location, a plurality of images of the environment, captured from viewpoints proximate the location, may be retrieved. For each location, the images may be reprojected to a three-dimensional shape and combined to generate a combined image. The combined image may be applied to one or more surfaces of the three-dimensional shape to generate a vantage. The vantages may be stored such that the vantages can be used to generate viewpoint video of the scene, as viewed from a virtual viewpoint corresponding to an actual viewer's viewpoint within the viewing volume.

公开(公告)号：US10567464B2

公开(公告)日：2020-02-18

申请号：US15832023

申请日：2017-12-05

Applicant: Google LLC

Inventor： Derek Pang ,  Colvin Pitts ,  Kurt Akeley

IPC: H04L29/06 ,  H04N19/124 ,  H04N19/44 ,  H04N19/196 ,  H04N19/13 ,  H04N5/247 ,  H04N13/156 ,  H04N5/225 ,  H04N13/243 ,  H04N5/222 ,  H04N13/194 ,  H04N13/117 ,  H04N19/176 ,  H04N19/33 ,  H04N13/161 ,  H04N13/282 ,  H04N13/232 ,  H04N19/103 ,  H04N19/147 ,  H04N19/46 ,  H04N19/86 ,  H04N19/597 ,  H04N13/349 ,  H04N19/553 ,  H04N13/275 ,  H04N21/81 ,  H04N13/344 ,  H04N19/436

Abstract: A video stream of a scene for a virtual reality or augmented reality experience may be captured by one or more image capture devices. Data from the video stream may be retrieved, including base vantage data with base vantage color data depicting the scene from a base vantage location, and target vantage data with target vantage color data depicting the scene from a target vantage location. The base vantage data may be reprojected to the target vantage location to obtain reprojected target vantage data. The reprojected target vantage data may be compared with the target vantage data to obtain residual data. The residual data may be compressed by removing a subset of the residual data that is likely to be less viewer-discernable than a remainder of the residual data. A compressed video stream may be stored, including the base vantage data and the compressed residual data.

公开(公告)号：US10474227B2

公开(公告)日：2019-11-12

申请号：US15897994

申请日：2018-02-15

Applicant: Google LLC

Inventor： Trevor Carothers ,  Kurt Akeley

IPC: G06F3/01 ,  G06T19/00 ,  H04N13/117 ,  G06T7/536 ,  H04N13/232 ,  H04N13/344 ,  H04N13/366

Abstract: A virtual reality or augmented reality experience may be presented for a viewer through the use of input including only three degrees of freedom. The input may include orientation data indicative of a viewer orientation at which a head of the viewer is oriented. The viewer orientation may be mapped to an estimated viewer location. Viewpoint video may be generated of a scene as viewed from a virtual viewpoint with a virtual location corresponding to the estimated viewer location, from along the viewer orientation. The viewpoint video may be displayed for the viewer. In some embodiments, mapping may be carried out by defining a ray at the viewer orientation, locating an intersection of the ray with a three-dimensional shape, and, based on a location of the intersection, generating the estimated viewer location. The shape may be generated via calibration with a device that receives input including six degrees of freedom.

公开(公告)号：US10469873B2

公开(公告)日：2019-11-05

申请号：US15730096

申请日：2017-10-11

Applicant: Google LLC

Inventor： Derek Pang ,  Colvin Pitts ,  Kurt Akeley ,  Zeyar Htet

IPC: H04N19/597 ,  H04N19/44 ,  H04N19/162 ,  H04N19/436 ,  H04N19/186 ,  H04N19/176 ,  H04N5/222 ,  H04N5/225 ,  H04N5/247 ,  H04N13/243 ,  H04N13/194 ,  H04N13/232 ,  H04N13/282 ,  H04N13/156 ,  H04N13/117 ,  H04N13/161 ,  H04N19/93 ,  H04N19/91 ,  H04N13/366 ,  H04N13/344 ,  H04N13/275

Abstract: A virtual reality or augmented reality experience of a scene may be decoded for playback for a viewer through a combination of CPU and GPU processing. A video stream may be retrieved from a data store. A first viewer position and/or orientation may be received from an input device, such as the sensor package on a head-mounted display (HMD). At a processor, the video stream may be partially decoded to generate a partially-decoded bitstream. At a graphics processor, the partially-decoded bitstream may be further decoded to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The viewpoint video may be displayed on a display device, such as screen of the HMD.

公开(公告)号：US10540818B2

公开(公告)日：2020-01-21

申请号：US15730481

申请日：2017-10-11

Applicant: Google LLC

Inventor： Kurt Akeley

IPC: G06T19/00 ,  G06T19/20 ,  H04N13/106 ,  H04N13/00

Abstract: Video data of an environment may be prepared for stereoscopic presentation to a user in a virtual reality or augmented reality experience. According to one method, a plurality of locations distributed throughout a viewing volume may be designated, at which a plurality of vantages are to be positioned to facilitate viewing of the environment from proximate the locations. For each location, a plurality of images of the environment, captured from viewpoints proximate the location, may be retrieved. For each location, the images may be reprojected to a three-dimensional shape and combined to generate a combined image. The combined image may be applied to one or more surfaces of the three-dimensional shape to generate a vantage. The vantages may be stored such that the vantages can be used to generate stereoscopic viewpoint video of the scene, as viewed from at least two virtual viewpoints corresponding to viewpoints of an actual viewer's eyes within the viewing volume.