-
公开(公告)号:US11810313B2
公开(公告)日:2023-11-07
申请号:US17249095
申请日:2021-02-19
申请人: GOOGLE LLC
发明人: Vladimir Tankovich , Christian Haene , Sean Ryan Francesco Fanello , Yinda Zhang , Shahram Izadi , Sofien Bouaziz , Adarsh Prakash Murthy Kowdle , Sameh Khamis
CPC分类号: G06T7/593 , G06T3/0093 , G06T3/40 , G06T5/30 , H04N13/20 , G06T2207/20016 , G06T2207/20084 , H04N2013/0081
摘要: According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.
-
公开(公告)号:US20220405569A1
公开(公告)日:2022-12-22
申请号:US17304505
申请日:2021-06-22
申请人: Google LLC
摘要: A method including, in a training phase, training a gaze prediction model including a first model and a second model, the first model and the second model being configured in conjunction to predict segmentation data based on training data, training a third model together with the first model and the second model, the third model being configured to predict a training characteristic using an output of the first model based on the training data, and in an operational phase, receiving operational data and predicting an operational characteristic using the trained first model and the trained third model.
-
公开(公告)号:US20240303908A1
公开(公告)日:2024-09-12
申请号:US18547628
申请日:2021-04-30
申请人: GOOGLE LLC
发明人: Yinda Zhang , Danhang Tang , Ruofei Du , Zhang Chen , Kyle Genova , Sofien Bouaziz , Thomas Allen Funkhouser , Sean Ryan Francesco Fanello , Christian Haene
摘要: A method including generating a first vector based on a first grid and a three-dimensional (3D) position associated with a first implicit representation (IR) of a 3D object, generating at least one second vector based on at least one second grid and an upsampled first grid, decoding the first vector to generate a second IR of the 3D object, decoding the at least one second vector to generate at least one third IR of the 3D object, generating a composite IR of the 3D object based on the second IR of the 3D object and the at least one third IR of the 3D object, and generating a reconstructed volume representing the 3D object based on the composite IR of the 3D object.
-
公开(公告)号:US20240212106A1
公开(公告)日:2024-06-27
申请号:US18554960
申请日:2021-04-28
申请人: Google LLC
发明人: Chloe LeGendre , Paul Debevec , Sean Ryan Francesco Fanello , Rohit Kumar Pandey , Sergio Orts Escolano , Christian Haene , Sofien Bouaziz
摘要: Apparatus and methods related to applying lighting models to images are provided. An example method includes receiving, via a computing device, an image comprising a subject. The method further includes relighting, via a neural network, a foreground of the image to maintain a consistent lighting of the foreground with a target illumination. The relighting is based on a per-pixel light representation indicative of a surface geometry of the foreground. The light representation includes a specular component, and a diffuse component, of surface reflection. The method additionally includes predicting, via the neural network, an output image comprising the subject in the relit foreground. One or more neural networks can be trained to perform one or more of the aforementioned aspects.
-
公开(公告)号:US11954899B2
公开(公告)日:2024-04-09
申请号:US18274371
申请日:2021-03-11
申请人: Google LLC
发明人: Yinda Zhang , Feitong Tan , Danhang Tang , Mingsong Dou , Kaiwen Guo , Sean Ryan Francesco Fanello , Sofien Bouaziz , Cem Keskin , Ruofei Du , Rohit Kumar Pandey , Deqing Sun
IPC分类号: G06V10/771 , G06T7/70 , G06T17/00 , G06V10/44 , G06V10/75
CPC分类号: G06V10/771 , G06T7/70 , G06T17/00 , G06V10/44 , G06V10/751 , G06T2207/20081 , G06T2207/20084
摘要: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.
-
6.
公开(公告)号:US20240046618A1
公开(公告)日:2024-02-08
申请号:US18274371
申请日:2021-03-11
申请人: Google LLC
发明人: Yinda Zhang , Feitong Tan , Danhang Tang , Mingsong Dou , Kaiwen Guo , Sean Ryan Francesco Fanello , Sofien Bouaziz , Cem Keskin , Ruofei Du , Rohit Kumar Pandey , Deqing Sun
IPC分类号: G06V10/771 , G06T17/00 , G06T7/70 , G06V10/44 , G06V10/75
CPC分类号: G06V10/771 , G06T17/00 , G06T7/70 , G06V10/44 , G06V10/751 , G06T2207/20081 , G06T2207/20084
摘要: Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.
-
公开(公告)号:US20220014723A1
公开(公告)日:2022-01-13
申请号:US17309440
申请日:2019-12-02
申请人: Google LLC
发明人: Rohit Pandey , Jonathan Taylor , Ricardo Martin Brualla , Shuoran Yang , Pavlo Pidlypenskyi , Daniel Goldman , Sean Ryan Francesco Fanello
IPC分类号: H04N13/111 , H04N13/161 , H04N13/366 , H04N13/243 , H04N13/332 , G06T7/174 , G06K9/32 , G06K9/00 , G06K9/46
摘要: Three-dimensional (3D) performance capture and machine learning can be used to re-render high quality novel viewpoints of a captured scene. A textured 3D reconstruction is first rendered to a novel viewpoint. Due to imperfections in geometry and low-resolution texture, the 2D rendered image contains artifacts and is low quality. Accordingly, a deep learning technique is disclosed that takes these images as input and generates more visually enhanced re-rendering. The system is specifically designed for VR and AR headsets, and accounts for consistency between two stereo views.
-
8.
公开(公告)号:US10997457B2
公开(公告)日:2021-05-04
申请号:US16616235
申请日:2019-10-16
申请人: Google LLC
发明人: Christoph Rhemann , Abhimitra Meka , Matthew Whalen , Jessica Lynn Busch , Sofien Bouaziz , Geoffrey Douglas Harvey , Andrea Tagliasacchi , Jonathan Taylor , Paul Debevec , Peter Joseph Denny , Sean Ryan Francesco Fanello , Graham Fyffe , Jason Angelo Dourgarian , Xueming Yu , Adarsh Prakash Murthy Kowdle , Julien Pascal Christophe Valentin , Peter Christopher Lincoln , Rohit Kumar Pandey , Christian Häne , Shahram Izadi
摘要: Methods, systems, and media for relighting images using predicted deep reflectance fields are provided. In some embodiments, the method comprises: identifying a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject; training a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises: for each training iteration in a series of training iterations and for each training sample in the group of training samples: generating an output predicted image, wherein the output predicted image is a representation of the subject associated with the training sample with lighting from the lighting direction associated with the training sample; identifying a ground-truth OLAT image included in the group of OLAT images for the training sample that corresponds to the lighting direction for the training sample; calculating a loss that indicates a perceptual difference between the output predicted image and the identified ground-truth OLAT image; and updating parameters of the convolutional neural network based on the calculated loss; identifying a test sample that includes a second group of spherical color gradient images and a second lighting direction; and generating a relit image of the subject included in each of the second group of spherical color gradient images with lighting from the second lighting direction using the trained convolutional neural network.
-
公开(公告)号:US20230154051A1
公开(公告)日:2023-05-18
申请号:US17919460
申请日:2020-04-17
申请人: Google LLC
发明人: Danhang Tang , Saurabh Singh , Cem Keskin , Phillip Andrew Chou , Christian Haene , Mingsong Dou , Sean Ryan Francesco Fanello , Jonathan Taylor , Andrea Tagliasacchi , Philip Lindsley Davidson , Yinda Zhang , Onur Gonen Guleryuz , Shahram Izadi , Sofien Bouaziz
IPC分类号: G06T9/00
摘要: Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.
-
公开(公告)号:US11328486B2
公开(公告)日:2022-05-10
申请号:US16861530
申请日:2020-04-29
申请人: Google LLC
发明人: Anastasia Tkach , Ricardo Martin Brualla , Shahram Izadi , Shuoran Yang , Cem Keskin , Sean Ryan Francesco Fanello , Philip Davidson , Jonathan Taylor , Rohit Pandey , Andrea Tagliasacchi , Pavlo Pidlypenskyi
摘要: A method includes receiving a first image including color data and depth data, determining a viewpoint associated with an augmented reality (AR) and/or virtual reality (VR) display displaying a second image, receiving at least one calibration image including an object in the first image, the object being in a different pose as compared to a pose of the object in the first image, and generating the second image based on the first image, the viewpoint and the at least one calibration image.
-
-
-
-
-
-
-
-
-