公开(公告)号：US20240087266A1

公开(公告)日：2024-03-14

申请号：US17973295

申请日：2022-10-25

Applicant: Snap Inc.

Inventor： Riza Alp Guler ,  Himmy Tam ,  Haoyang Wang ,  Antonios Kakolyris

IPC: G06T19/20 ,  G06T3/00 ,  G06T7/10 ,  G06T19/00

CPC classification number: G06T19/20 ,  G06T3/0093 ,  G06T7/10 ,  G06T19/006 ,  G06T2200/04 ,  G06T2207/10016 ,  G06T2207/20021 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30196 ,  G06T2210/22 ,  G06T2219/2021

Abstract: Methods and systems are disclosed for performing real-time deforming operations. The system receives an image that includes a depiction of a real-world object. The system applies a machine learning model to the image to generate a warping field and segmentation mask, the machine learning model trained to establish a relationship between a plurality of training images depicting real-world objects and corresponding ground-truth warping fields and segmentation masks associated with a target shape. The system applies the generated warping field and segmentation mask to the image to warp the real-world object depicted in the image to the target shape.

公开(公告)号：US11816850B2

公开(公告)日：2023-11-14

申请号：US17301926

申请日：2021-04-19

Applicant: Snap Inc.

Inventor： Riza Alp Guler ,  Iason Kokkinos

IPC: G06T7/73 ,  G06T17/00 ,  G06T19/20 ,  G06T7/50 ,  G06N3/02 ,  G06V20/64 ,  G06V10/764 ,  G06V10/82 ,  G06V10/44 ,  G06V10/22 ,  G06V40/10

CPC classification number: G06T7/50 ,  G06N3/02 ,  G06T7/75 ,  G06T17/005 ,  G06T19/20 ,  G06V10/225 ,  G06V10/454 ,  G06V10/764 ,  G06V10/82 ,  G06V20/64 ,  G06V40/103 ,  G06T2207/20084 ,  G06T2207/20221 ,  G06T2207/30196 ,  G06T2219/2004

Abstract: This disclosure relates to reconstructing three-dimensional models of objects from two-dimensional images. According to a first aspect, this specification describes a computer implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying a type of said object; spatially anchoring a pre-determined set of object landmarks within the image; extracting a two-dimensional image representation from each object landmark; estimating a respective three-dimensional representation for the respective two-dimensional image representations; and combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

公开(公告)号：US20230316666A1

公开(公告)日：2023-10-05

申请号：US17842006

申请日：2022-06-16

Applicant: Snap Inc.

Inventor： Madiyar Aitbayev ,  Brian Fulkerson ,  Riza Alp Guler ,  Georgios Papandreou ,  Himmy Tam

IPC: G06T19/00 ,  G06V10/70 ,  G06V10/26 ,  G06V20/20 ,  G06N20/20

CPC classification number: G06T19/006 ,  G06V10/70 ,  G06V10/26 ,  G06V20/20 ,  G06N20/20

Abstract: Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; extracting a portion of the image; applying a first machine learning model stage to the portion to predict a depth of a point of interest for the data representing the depiction of the person; applying a second machine learning model stage to the portion of the image to predict a relative depth of each pixel in the portion of the image to the predicted depth of the point of interest; generating dense depth reconstruction of the data representing the depiction of the person based on outputs of the first and second stages of the machine learning model; and applying one or more AR elements to the image based on the dense depth reconstruction.

公开(公告)号：US20230316665A1

公开(公告)日：2023-10-05

申请号：US17841994

申请日：2022-06-16

Applicant: Snap Inc.

Inventor： Madiyar Aitbayev ,  Brian Fulkerson ,  Riza Alp Guler ,  Georgios Papandreou ,  Himmy Tam

IPC: G06T19/00 ,  G06T7/11 ,  G06T7/194 ,  G06T15/50 ,  G06T7/70 ,  G06T19/20 ,  G06T15/60

CPC classification number: G06T19/006 ,  G06T7/11 ,  G06T7/194 ,  G06T15/506 ,  G06T7/70 ,  G06T19/20 ,  G06T15/60 ,  G06T2207/30196 ,  G06T2207/20021 ,  G06T2207/20081 ,  G06T2207/10016 ,  G06T2219/2012 ,  G06T2207/20084 ,  G06T2200/08 ,  G06T2210/16

Abstract: Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; generating a segmentation of the data representing the person depicted in the image; extracting a portion of the image corresponding to the segmentation of the data representing the person depicted in the image; applying a machine learning model to the portion of the image to predict a surface normal tensor for the data representing the depiction of the person, the surface normal tensor representing surface normals of each pixel within the portion of the image; and applying one or more augmented reality (AR) elements to the image based on the surface normal tensor.

公开(公告)号：US20230090645A1

公开(公告)日：2023-03-23

申请号：US17448158

申请日：2021-09-20

Applicant: Snap Inc

Inventor： Matan Zohar ,  Yanli Zhao ,  Brian Fulkerson ,  Riza Alp Guler

IPC: G06T19/00 ,  G06T7/20 ,  G06T17/20 ,  G06T7/60 ,  G06T11/00

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; determining UV positions of the real-world object depicted in the video to obtain pixel values associated with the UV positions; generating an external mesh and associated augmented reality (AR) element representing the real-world object based on the pixel values associated with the UV positions; deforming the external mesh based on changes to the 3D body mesh and a deformation parameter; and modifying the video to replace the real-world object with the AR element based on the deformed external mesh.

公开(公告)号：US20230070008A1

公开(公告)日：2023-03-09

申请号：US17760424

申请日：2020-02-17

Applicant: Snap Inc.

Inventor： Dominik Kulon ,  Riza Alp Guler ,  lason Kokkinos ,  Stefanos Zafeiriou

IPC: G06T17/20 ,  G06T7/00 ,  G06T3/40

Abstract: This specification discloses methods and systems for generating three-dimensional models of deformable objects from two-dimensional images. According to one aspect of this disclosure, there is described a computer implemented method for generating a three dimensional model of deformable object from a two-dimensional image. The method comprises: receiving, as input to an embedding neural network, the two-dimensional image, wherein the two dimensional image comprises an image of an object; generating, using the embedding neural network, an embedded representation of a two-dimensional image; inputting the embedded representation into a learned decoder model; and generating, using the learned decoder model, parameters of the three dimensional model of the object from the embedded representation.