公开(公告)号：US20240265498A1

公开(公告)日：2024-08-08

申请号：US18164458

申请日：2023-02-03

Applicant: Snap inc.

Inventor： Aleksandr Belskikh ,  Georgii Grigorev ,  Pavel Savchenkov

IPC: G06T5/50 ,  G06T3/40 ,  G06T5/00

CPC classification number: G06T5/50 ,  G06T3/4046 ,  G06T5/70 ,  G06T2200/24 ,  G06T2207/20084 ,  G06T2207/20221 ,  G06T2207/30201

Abstract: The subject technology receives an input image and a segmentation mask of the input image. The subject technology obtains reconstructed noise of the input image using the input image and the segmentation mask. The subject technology determines a first set of features by performing a first portion of a forward pass of the reconstructed noise through a decoder. The subject technology determines a second set of features by processing the input image for stable diffusion using an image to image (IMG2IMG) model. The subject technology generates a third set of features based on combining, using the segmentation mask, the first set of features and the second set of features with the reconstructed noise. The subject technology generates an output image by performing a remaining portion of the forward pass of the third set of features through the decoder.

公开(公告)号：US20240221259A1

公开(公告)日：2024-07-04

申请号：US18149007

申请日：2022-12-30

Applicant: Snap Inc.

Inventor： Aleksandr Belskikh ,  Menglei Chai ,  Antoine Chassang ,  Anna Kovalenko ,  Pavel Savchenkov

IPC: G06T13/40 ,  G06N3/094 ,  G06T19/00

CPC classification number: G06T13/40 ,  G06N3/094 ,  G06T19/006

Abstract: The subject technology generates a first image of a face using a GAN model. The subject technology applies 3D virtual hair on the first image to generate a second image with 3D virtual hair. The subject technology projects the second image with 3D virtual hair into a GAN latent space to generate a third image with realistic virtual hair. The subject technology performs a blend of the realistic virtual hair with the first image of the face to generate a new image with new realistic hair that corresponds to the 3D virtual hair. The subject technology trains a neural network that receives the second image with the 3D virtual hair and provides an output image with realistic virtual hair. The subject technology generates using the trained neural network, a particular output image with realistic hair based on a particular input image with 3D virtual hair.

公开(公告)号：US20230351998A1

公开(公告)日：2023-11-02

申请号：US18218028

申请日：2023-07-04

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Maxim Lukin ,  Aleksandr Mashrabov

IPC: G10L13/00 ,  G10L13/08 ,  G06T13/40 ,  G06V40/16 ,  G06V10/764 ,  G06V10/82

CPC classification number: G10L13/00 ,  G10L13/08 ,  G06T13/40 ,  G06V40/171 ,  G06V10/764 ,  G06V10/82

Abstract: Systems and methods for text and audio-based real-time face reenactment are provided. An example method includes receiving an input text and a target image, where the target image including a target face, generating, based on the input text, a sequence of sets of acoustic features corresponding to the input text, generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points, generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points, determining, based on the set sequence of sets of facial key points, a sequence of deformations of the target face, and applying the sequence of deformations to the target image, thereby generating a sequence of frames of an output video.

公开(公告)号：US20220392133A1

公开(公告)日：2022-12-08

申请号：US17881947

申请日：2022-08-05

Applicant: Snap Inc.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06T7/70 ,  G06V40/16

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.

公开(公告)号：US20220319231A1

公开(公告)日：2022-10-06

申请号：US17709711

申请日：2022-03-31

Applicant: Snap Inc.

Inventor： Alexey Pankov ,  Pavel Savchenkov

IPC: G06V40/16 ,  G06T17/20 ,  G06T19/20 ,  G06T7/10 ,  G06V10/82 ,  G06T3/00

Abstract: The subject technology receives frames of a source media content, the frames of the source media content including representations of a head and a face of a source actor. The subject technology generates, based at least in part on the frames of the source media content, sets of source pose parameters. The subject technology receives at least one target image, the at least one target image including representations of a target head and a target face of a target entity. The subject technology provides the sets of source pose parameters to a neural network to determine facial landmarks for head turns and facial expressions. The subject technology generates, based at least in part on the sets of source pose parameters and the facial landmarks for head turns and facial expressions, an output media content. The subject technology provides augmented reality content based at least in part on the output media content for display on a computing device.

公开(公告)号：US20220270332A1

公开(公告)日：2022-08-25

申请号：US17663176

申请日：2022-05-12

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06N20/00 ,  G06T13/40 ,  H04L51/046

Abstract: A methodology for training a machine learning model to generate color-neutral input face images is described. For each training face image from a training dataset that is used for training the model, the training system generates an input face image, which has the color and lighting of a randomly selected image from the set of color source images, and which has facial features and expression of a face object from the training face image. Because, during training, the machine learning model is “confused” by changing the color and lighting of a training face image to a randomly selected different color and lighting, the trained machine learning model generates a color neutral embedding representing facial features from the training face image.

公开(公告)号：US11335069B1

公开(公告)日：2022-05-17

申请号：US17107410

申请日：2020-11-30

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Yurii Volkov ,  Jeremy Baker Voss

IPC: G06T19/00 ,  G06T13/40 ,  G06N20/00 ,  H04L51/046

Abstract: In some embodiments, users' experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actor's face in the frames of a video with a user's face from the user's portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actor's face but, at the same time, has the likeness of the user's face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.

公开(公告)号：US20210327404A1

公开(公告)日：2021-10-21

申请号：US17355834

申请日：2021-06-23

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Maxim Lukin ,  Aleksandr Mashrabov

IPC: G10L13/00 ,  G06K9/00 ,  G10L13/08 ,  G06T13/40

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.

公开(公告)号：US10789453B2

公开(公告)日：2020-09-29

申请号：US16251436

申请日：2019-01-18

Applicant: SNAP INC.

Inventor： Pavel Savchenkov ,  Dmitry Matov ,  Aleksandr Mashrabov ,  Alexey Pchelnikov

IPC: G06K9/00 ,  G06T11/00 ,  G06Q30/02 ,  G06N3/04

Abstract: Provided are systems and a method for photorealistic real-time face reenactment. An example method includes receiving a target video including a target face and a source video including a source face. The method includes determining, based on the target face, a target facial expression. The method includes determining, based on the source face, a source facial expression. The method includes synthesizing, using the parametric face model, an output face. The output face including the target face wherein the target facial expression is modified to imitate the source facial expression. The method includes generating, based on a deep neural network, mouth and eyes regions, and combining the output face, the mouth, and eyes regions to generate a frame of an output video.

公开(公告)号：US20200234480A1

公开(公告)日：2020-07-23

申请号：US16662743

申请日：2019-10-24

Applicant: SNAP INC.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Maxim Lukin ,  Ivan Belonogov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06K9/00 ,  G06T7/70

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.