公开(公告)号：WO2022115325A1

公开(公告)日：2022-06-02

申请号：PCT/US2021/060079

申请日：2021-11-19

Applicant: SNAP INC.

Inventor： SAVCHENKOV, Pavel ,  VOLKOV, Yurii ,  VOSS, Jeremy Baker

IPC: G06T11/00

Abstract: In some embodiments, users experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actors face in the frames of a video with a users face from the users portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actors face but, at the same time, has the likeness of the users face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.

公开(公告)号：WO2020150689A1

公开(公告)日：2020-07-23

申请号：PCT/US2020/014222

申请日：2020-01-18

Applicant: SNAP INC.

Inventor： VOLKOV, Yurii ,  SAVCHENKOV, Pavel ,  LUKIN, Maxim ,  BELONOGOV, Ivan ,  SMIRNOV, Nikolai ,  MASHRABOV, Aleksandr

IPC: G06K9/62 ,  G06K9/46 ,  G06K9/00

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.

公开(公告)号：WO2020150687A1

公开(公告)日：2020-07-23

申请号：PCT/US2020/014220

申请日：2020-01-18

Applicant: SNAP INC.

Inventor： KROKHALEV, Eugene ,  MASHRABOV, Aleksandr ,  SAVCHENKOV, Pavel

IPC: G06T13/40 ,  G06T13/80

Abstract: Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face. The method further includes receiving a target image with a second face. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations, wherein the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

公开(公告)号：WO2022213099A1

公开(公告)日：2022-10-06

申请号：PCT/US2022/071456

申请日：2022-03-31

Applicant: SNAP INC. ,  PANKOV, Alexey ,  SAVCHENKOV, Pavel

Inventor： PANKOV, Alexey ,  SAVCHENKOV, Pavel

IPC: G06T17/00 ,  G06T19/20 ,  G06T17/20 ,  G06T2219/2004 ,  G06T2219/2016 ,  G06T2219/2021 ,  G06T3/0093 ,  G06T7/10 ,  G06V10/82 ,  G06V2201/07 ,  G06V40/168

Abstract: The subject technology receives frames of a source media content, the frames of the source media content including representations of a head and a face of a source actor. The subject technology generates, based at least in part on the frames of the source media content, sets of source pose parameters. The subject technology receives at least one target image, the at least one target image including representations of a target head and a target face of a target entity. The subject technology provides the sets of source pose parameters to a neural network to determine facial landmarks for head turns and facial expressions. The subject technology generates, based at least in part on the sets of source pose parameters and the facial landmarks for head turns and facial expressions, an output media content. The subject technology provides augmented reality content based at least in part on the output media content for display on a computing device.

公开(公告)号：WO2020150692A1

公开(公告)日：2020-07-23

申请号：PCT/US2020/014225

申请日：2020-01-18

Applicant: SNAP INC.

Inventor： SHABUROV, Victor ,  SAVCHENKOV, Pavel ,  MASHRABOV, Alexander ,  MATOV, Dmitry ,  SAVINOVA, Sofia ,  PCHELNIKOV, Alexey ,  GOLOBKOV, Roman

IPC: G11B27/031 ,  G06T11/60

Abstract: Disclosed are systems and methods for template-based generation of personalized videos. An example method may commence with receiving video configuration data including a sequence of frame images, a sequence of face area parameters defining positions of a face area in the frame images, and a sequence of facial landmark parameters defining positions of facial landmarks in the frame images. The method may continue with receiving an image of a source face. The method may further include generating an output video. The generation of the output video may include modifying a frame image of the sequence of frame images. Specifically, the image of the source face may be modified to obtain a further image featuring the source face adopting a facial expression corresponding to the facial landmark parameters. The further image may be inserted into the frame image at position determined by face area parameters corresponding to the frame image.

公开(公告)号：WO2020150688A1

公开(公告)日：2020-07-23

申请号：PCT/US2020/014221

申请日：2020-01-18

Applicant: SNAP INC.

Inventor： SAVCHENKOV, Pavel ,  LUKIN, Maxim ,  MASHRABOV, Aleksandr

IPC: G10L13/04 ,  G06K9/00 ,  G10L13/08

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; determining, based on the sequence of sets of acoustic features, a sequence of sets of scenario data indicating modifications of the target face for pronouncing the input text; generating, based on the sequence of sets of scenario data, a sequence of frames, wherein each of the frames includes the target face modified based on at least one of the sets of scenario data; generating, based on the sequence of frames, an output video; and synthesizing, based on the sequence of sets of acoustic features, an audio data and adding the audio data to the output video.

公开(公告)号：WO2020150686A1

公开(公告)日：2020-07-23

申请号：PCT/US2020/014219

申请日：2020-01-18

Applicant: SNAP INC.

Inventor： SAVCHENKOV, Pavel ,  MATOV, Dmitry ,  MASHRABOV, Aleksandr ,  PCHELNIKOV, Alexey

IPC: G06K9/00 ,  G06T11/00

Abstract: Provided are systems and a method for photorealistic real-time face reenactment. An example method includes receiving a target video including a target face and a source video including a source face. The method includes determining, based on the target face, a target facial expression. The method includes determining, based on the source face, a source facial expression. The method includes synthesizing, using the parametric face model, an output face. The output face including the target face wherein the target facial expression is modified to imitate the source facial expression. The method includes generating, based on a deep neural network, mouth and eyes regions, and combining the output face, the mouth, and eyes regions to generate a frame of an output video.