-
公开(公告)号：US20240420407A1
公开(公告)日：2024-12-19
申请号：US18211149
申请日：2023-06-16
Applicant: Snap Inc.
Inventor： Evangelos Ntavelis , Kyle Olszewski , Aliaksandr Siarohin , Sergey Tulyakov
Abstract: Systems and methods for generating static and articulated 3D assets are provided that include a 3D autodecoder at their core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation for rendering view-consistent appearance and geometry. The appropriate intermediate volumetric latent space is then identified and robust normalization and de-normalization operations are implemented to learn a 3D diffusion from 2D images or monocular videos of rigid or articulated objects. The methods are flexible enough to use either existing camera supervision or no camera information at all—instead efficiently learning the camera information during training. The generated results are shown to outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and a large-scale, real video dataset of static objects.
-
公开(公告)号：US20240029346A1
公开(公告)日：2024-01-25
申请号：US17814063
申请日：2022-07-21
Applicant: Snap Inc.
Inventor： Zeng Huang , Menglei Chai , Sergey Tulyakov , Kyle Olszewski , Hsin-Ying Lee
CPC classification number: G06T17/00 , G06T15/04 , G06T2207/10028
Abstract: A system to enable 3D hair reconstruction and rendering from a single reference image which performs a multi-stage process that utilizes both a 3D implicit representation and a 2D parametric embedding space.
-
公开(公告)号：US20230215085A1
公开(公告)日：2023-07-06
申请号：US18090091
申请日：2022-12-28
Applicant: Snap Inc.
Inventor： Kyle Olszewski , Sergey Tulyakov , Zhengfei Kuang , Menglei Chai
CPC classification number: G06T15/50 , G06T7/60 , G06T7/80 , G06T15/06 , G06T7/55 , G06T7/194 , G06T2210/12 , G06T2207/10028 , G06T2207/20084 , G06T2207/20081
Abstract: Three-dimensional object representation and re-rendering systems and methods for producing a 3D representation of an object from 2D images including the object that enables object-centric rendering. A modular approach is used that optimizes a Neural Radiance Field (NeRF) model to estimate object geometry and refine camera parameters and, then, infer surface material properties and per-image lighting conditions that fit the 2D images.
-
公开(公告)号：US20240273809A1
公开(公告)日：2024-08-15
申请号：US18644653
申请日：2024-04-24
Applicant: Snap Inc.
Inventor： Zeng Huang , Jian Ren , Sergey Tulyakov , Menglei Chai , Kyle Olszewski , Huan Wang
CPC classification number: G06T15/06 , G06T7/97 , G06T2207/20081 , G06T2207/20084
Abstract: Methods and systems are disclosed for performing operations for generating a 3D model of a scene. The operations include: receiving a set of two-dimensional (2D) images representing a first view of a real-world environment; applying a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value; and generating a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.
-
公开(公告)号：US12002146B2
公开(公告)日：2024-06-04
申请号：US17656778
申请日：2022-03-28
Applicant: Snap Inc.
Inventor： Zeng Huang , Jian Ren , Sergey Tulyakov , Menglei Chai , Kyle Olszewski , Huan Wang
CPC classification number: G06T15/06 , G06T7/97 , G06T2207/20081 , G06T2207/20084
Abstract: Methods and systems are disclosed for performing operations for generating a 3D model of a scene. The operations include: receiving a set of two-dimensional (2D) images representing a first view of a real-world environment; applying a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value; and generating a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.
-
Patent Agency Ranking
公开(公告)号：US20230386158A1
公开(公告)日：2023-11-30
申请号：US17814391
申请日：2022-07-22
Applicant: Snap Inc.
Inventor： Menglei Chai , Sergey Tulyakov , Jian Ren , Hsin-Ying Lee , Kyle Olszewski , Zeng Huang , Zezhou Cheng
CPC classification number: G06T19/20 , G06T17/00 , G06T2219/2012 , G06T2219/2021
Abstract: Systems, computer readable media, and methods herein describe an editing system where a three-dimensional (3D) object can be edited by editing a 2D sketch or 2D RGB views of the 3D object. The editing system uses multi-modal (MM) variational auto-decoders (VADs)(MM-VADs) that are trained with a shared latent space that enables editing 3D objects by editing 2D sketches of the 3D objects. The system determines a latent code that corresponds to an edited or sketched 2D sketch. The latent code is then used to generate a 3D object using the MM-VADs with the latent code as input. The latent space is divided into a latent space for shapes and a latent space for colors. The MM-VADs are trained with variational auto-encoders (VAE) and a ground truth.
-
公开(公告)号：US12056792B2
公开(公告)日：2024-08-06
申请号：US17557834
申请日：2021-12-21
Applicant: Snap Inc.
Inventor： Jian Ren , Menglei Chai , Oliver Woodford , Kyle Olszewski , Sergey Tulyakov
CPC classification number: G06T11/00 , G06N3/045 , G06T3/60 , G06T7/194 , G06V40/10 , G06T2207/20084 , G06T2207/30196
Abstract: Systems and methods herein describe a motion retargeting system. The motion retargeting system accesses a plurality of two-dimensional images comprising a person performing a plurality of body poses, extracts a plurality of implicit volumetric representations from the plurality of body poses, generates a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose, and based on the three-dimensional warping field, generates a two-dimensional image of an artificial person performing the target pose.
-
公开(公告)号：US20230306675A1
公开(公告)日：2023-09-28
申请号：US17656778
申请日：2022-03-28
Applicant: Snap Inc.
Inventor： Zeng Huang , Jian Ren , Sergey Tulyakov , Menglei Chai , Kyle Olszewski , Huan Wang
CPC classification number: G06T15/06 , G06T7/97 , G06T2207/20081 , G06T2207/20084
Abstract: Methods and systems are disclosed for performing operations for generating a 3D model of a scene. The operations include: receiving a set of two-dimensional (2D) images representing a first view of a real-world environment; applying a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value; and generating a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.
-
公开(公告)号：US20220207786A1
公开(公告)日：2022-06-30
申请号：US17557834
申请日：2021-12-21
Applicant: Snap Inc.
Inventor： Jian Ren , Menglei Chai , Oliver Woodford , Kyle Olszewski , Sergey Tulyakov
Abstract: Systems and methods herein describe a motion retargeting system. The motion retargeting system accesses a plurality of two-dimensional images comprising a person performing a plurality of body poses, extracts a plurality of implicit volumetric representations from the plurality of body poses, generates a three-dimensional warping field, the three-dimensional warping field configured to warp the plurality of implicit volumetric representations from a canonical pose to a target pose, and based on the three-dimensional warping field, generates a two-dimensional image of an artificial person performing the target pose.
-
公开(公告)号：US12094073B2
公开(公告)日：2024-09-17
申请号：US17814391
申请日：2022-07-22
Applicant: Snap Inc.
Inventor： Menglei Chai , Sergey Tulyakov , Jian Ren , Hsin-Ying Lee , Kyle Olszewski , Zeng Huang , Zezhou Cheng
CPC classification number: G06T19/20 , G06T17/00 , G06T2219/2012 , G06T2219/2021
Abstract: Systems, computer readable media, and methods herein describe an editing system where a three-dimensional (3D) object can be edited by editing a 2D sketch or 2D RGB views of the 3D object. The editing system uses multi-modal (MM) variational auto-decoders (VADs)(MM-VADs) that are trained with a shared latent space that enables editing 3D objects by editing 2D sketches of the 3D objects. The system determines a latent code that corresponds to an edited or sketched 2D sketch. The latent code is then used to generate a 3D object using the MM-VADs with the latent code as input. The latent space is divided into a latent space for shapes and a latent space for colors. The MM-VADs are trained with variational auto-encoders (VAE) and a ground truth.
-
-
-
-
-
-
-
-
-
Search Results
11
records
(took 0.015 seconds)
