公开(公告)号：US12079927B2

公开(公告)日：2024-09-03

申请号：US17506248

申请日：2021-10-20

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T15/50 ,  G06N3/08 ,  G06V10/60

CPC classification number: G06T15/506 ,  G06N3/08 ,  G06V10/60

Abstract: A messaging system performs image processing to estimate lighting properties with neural networks for images provided by users of the messaging system. A method of estimating light properties includes receiving an input image with first lighting properties and processing the input image using a convolutional neural network to generate an estimate of the first lighting properties. The method may further include modifying the input image with an augmentation to generate a modified input image, where the augmentation has second lighting properties, and changing the second lighting properties of the augmentation in the modified input image to the estimate of the first lighting properties.

公开(公告)号：US11410364B2

公开(公告)日：2022-08-09

申请号：US16662743

申请日：2019-10-24

Applicant: SNAP INC.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Maxim Lukin ,  Ivan Belonogov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06T7/70 ,  G06V40/16

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.

公开(公告)号：US12211166B2

公开(公告)日：2025-01-28

申请号：US18386515

申请日：2023-11-02

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T19/20 ,  G06N3/02 ,  G06N3/047 ,  G06N3/0475 ,  G06N3/0895 ,  G06N3/094 ,  G06T17/20 ,  H04L51/10

Abstract: A messaging system processes three-dimensional (3D) models to generate ground truths for training machine learning models for applications of the messaging system. A method of generating ground truths for machine learning includes generating a plurality of first rendered images from a first 3D base model where each first rendered image includes the 3D base model modified by first augmentations of a plurality of augmentations. The method further includes determining for a second 3D base model incompatible augmentations of the first plurality of augmentations, where the incompatible augmentations indicate changes to fixed features of the second 3D base model, and generating a plurality of second rendered images from a second 3D base model, each second rendered image comprising the second 3D base model modified by second augmentations, the second augmentations corresponding to the first augmentations of a corresponding first rendered image, where the second augmentations comprises augmentations of the first augmentations that are not incompatible augmentations.

公开(公告)号：US11915355B2

公开(公告)日：2024-02-27

申请号：US17881947

申请日：2022-08-05

Applicant: Snap Inc.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06T7/70 ,  G06V40/16

CPC classification number: G06T13/40 ,  G06T7/70 ,  G06V40/165 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20132

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.

公开(公告)号：US11847756B2

公开(公告)日：2023-12-19

申请号：US17506215

申请日：2021-10-20

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T19/20 ,  G06N3/02 ,  G06T17/20 ,  H04L51/10

CPC classification number: G06T19/20 ,  G06N3/02 ,  G06T17/205 ,  H04L51/10

Abstract: A messaging system processes three-dimensional (3D) models to generate ground truths for training machine learning models for applications of the messaging system. A method of generating ground truths for machine learning includes generating a plurality of first rendered images from a first 3D base model where each first rendered image includes the 3D base model modified by first augmentations of a plurality of augmentations. The method further includes determining for a second 3D base model incompatible augmentations of the first plurality of augmentations, where the incompatible augmentations indicate changes to fixed features of the second 3D base model, and generating a plurality of second rendered images from a second 3D base model, each second rendered image comprising the second 3D base model modified by second augmentations, the second augmentations corresponding to the first augmentations of a corresponding first rendered image, where the second augmentations comprises augmentations of the first augmentations that are not incompatible augmentations.

公开(公告)号：US20220207819A1

公开(公告)日：2022-06-30

申请号：US17506248

申请日：2021-10-20

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T15/50 ,  G06N3/08 ,  G06K9/46

Abstract: A messaging system performs image processing to estimate lighting properties with neural networks for images provided by users of the messaging system. A method of estimating light properties includes receiving an input image with first lighting properties and processing the input image using a convolutional neural network to generate an estimate of the first lighting properties. The method may further include modifying the input image with an augmentation to generate a modified input image, where the augmentation has second lighting properties, and changing the second lighting properties of the augmentation in the modified input image to the estimate of the first lighting properties.

公开(公告)号：US20240070976A1

公开(公告)日：2024-02-29

申请号：US18387212

申请日：2023-11-06

Applicant: Snap Inc.

Inventor： Yurii Volkov ,  Egor Nemchinov ,  Gleb Dmukhin

IPC: G06T15/50 ,  G06T7/73 ,  G06T19/20

CPC classification number: G06T15/503 ,  G06T7/75 ,  G06T15/506 ,  G06T19/20 ,  G06T2200/24 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2219/2012

Abstract: A messaging system performs image processing to relight objects with neural networks for images provided by users of the messaging system. A method of relighting objects with neural networks includes receiving an input image with first lighting properties comprising an object with second lighting properties and processing the input image using a convolutional neural network to generate an output image with the first lighting properties and comprising the object with third lighting properties, where the convolutional neural network is trained to modify the second lighting properties to be consistent with lighting conditions indicated by the first lighting properties to generate the third lighting properties. The method further includes modifying the second lighting properties of the object to generate the object with modified second lighting properties and blending the third lighting properties with the modified second lighting properties to generate a modified output image comprising the object with fourth lighting properties.

公开(公告)号：US20240062500A1

公开(公告)日：2024-02-22

申请号：US18386515

申请日：2023-11-02

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T19/20 ,  G06N3/02 ,  G06T17/20

CPC classification number: G06T19/20 ,  G06N3/02 ,  G06T17/205 ,  H04L51/10

Abstract: A messaging system processes three-dimensional (3D) models to generate ground truths for training machine learning models for applications of the messaging system. A method of generating ground truths for machine learning includes generating a plurality of first rendered images from a first 3D base model where each first rendered image includes the 3D base model modified by first augmentations of a plurality of augmentations. The method further includes determining for a second 3D base model incompatible augmentations of the first plurality of augmentations, where the incompatible augmentations indicate changes to fixed features of the second 3D base model, and generating a plurality of second rendered images from a second 3D base model, each second rendered image comprising the second 3D base model modified by second augmentations, the second augmentations corresponding to the first augmentations of a corresponding first rendered image, where the second augmentations comprises augmentations of the first augmentations that are not incompatible augmentations.

公开(公告)号：US20230118572A1

公开(公告)日：2023-04-20

申请号：US17506215

申请日：2021-10-20

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T19/20 ,  G06T17/20 ,  G06N3/02

Abstract: A messaging system processes three-dimensional (3D) models to generate ground truths for training machine learning models for applications of the messaging system. A method of generating ground truths for machine learning includes generating a plurality of first rendered images from a first 3D base model where each first rendered image includes the 3D base model modified by first augmentations of a plurality of augmentations. The method further includes determining for a second 3D base model incompatible augmentations of the first plurality of augmentations, where the incompatible augmentations indicate changes to fixed features of the second 3D base model, and generating a plurality of second rendered images from a second 3D base model, each second rendered image comprising the second 3D base model modified by second augmentations, the second augmentations corresponding to the first augmentations of a corresponding first rendered image, where the second augmentations comprises augmentations of the first augmentations that are not incompatible augmentations.

公开(公告)号：US20220172438A1

公开(公告)日：2022-06-02

申请号：US17107410

申请日：2020-11-30

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06T13/40 ,  H04L12/58 ,  G06N20/00

Abstract: In some embodiments, users' experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actor's face in the frames of a video with a user's face from the user's portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actor's face but, at the same time, has the likeness of the user's face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.