公开(公告)号：US11410364B2

公开(公告)日：2022-08-09

申请号：US16662743

申请日：2019-10-24

Applicant: SNAP INC.

Inventor： Yurii Volkov ,  Pavel Savchenkov ,  Maxim Lukin ,  Ivan Belonogov ,  Nikolai Smirnov ,  Aleksandr Mashrabov

IPC: G06T13/40 ,  G06T7/70 ,  G06V40/16

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.

公开(公告)号：US20210327117A1

公开(公告)日：2021-10-21

申请号：US17325225

申请日：2021-05-20

Applicant: Snap Inc.

Inventor： Eugene Krokhalev ,  Aleksandr Mashrabov ,  Pavel Savchenkov

IPC: G06T13/80 ,  G06T7/174 ,  G06K9/00

Abstract: Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face of a first person. The method further includes receiving a target image with a second face of a second person. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face and a background in the target image. The 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying the 2D deformations to the target image to obtain at least one output frame of an output video.

公开(公告)号：US20210012090A1

公开(公告)日：2021-01-14

申请号：US17034029

申请日：2020-09-28

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Dmitry Matov ,  Aleksandr Mashrabov ,  Alexey Pchelnikov

IPC: G06K9/00 ,  G06N3/04 ,  G06Q30/02 ,  G06T11/00

Abstract: Provided are systems and a method for photorealistic real-time face reenactment. An example method includes receiving a target video including a target face and a scenario including a series of source facial expressions, determining, based on the target face, one or more target facial expressions, and synthesizing, using the parametric face model, an output face. The output face includes the target face. The one or more target facial expressions are modified to imitate the source facial expressions. The method further includes generating, based on a deep neural network, a mouth region and an eyes region, and combining the output face, the mouth region, and the eyes region to generate a frame of an output video.

公开(公告)号：US12124803B2

公开(公告)日：2024-10-22

申请号：US17820437

申请日：2022-08-17

Applicant: Snap Inc.

Inventor： Arnab Ghosh ,  Jian Ren ,  Pavel Savchenkov ,  Sergey Tulyakov

IPC: G06F40/289 ,  G06F3/04842 ,  G06F16/583 ,  G06T11/60 ,  H04L51/10

CPC classification number: G06F40/289 ,  G06F3/04842 ,  G06F16/5846 ,  G06T11/60 ,  H04L51/10 ,  G06T2200/24

Abstract: A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The generated image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

公开(公告)号：US20240295953A1

公开(公告)日：2024-09-05

申请号：US18116003

申请日：2023-03-01

Applicant: Snap Inc.

Inventor： Aleksandr Zakharov ,  Sergey Smetanin ,  Arnab Ghosh ,  Pavel Savchenkov

IPC: G06F3/04845 ,  G06T11/00

CPC classification number: G06F3/04845 ,  G06T11/00 ,  G06F40/12 ,  G06T2200/24

Abstract: Examples disclosed herein describe prompt modification techniques for automated image generation. An image generation request comprising a base prompt is received from a user device. A plurality of prompt modifiers is identified. A processor-implemented scoring engine determines, for each prompt modifier, a modifier score. The modifier score for each prompt modifier is associated with the base prompt. One or more of the prompt modifiers are automatically selected based on the modifier scores. A modified prompt is generated. The modified prompt is based on the base prompt and the one or more selected prompt modifiers. The modified prompt is provided as input to an automated image generator to generate an image, and the image is caused to be presented on the user device.

公开(公告)号：US11995758B2

公开(公告)日：2024-05-28

申请号：US18080779

申请日：2022-12-14

Applicant: Snap Inc.

Inventor： Eugene Krokhalev ,  Aleksandr Mashrabov ,  Pavel Savchenkov

IPC: G06T13/80 ,  G06T7/174 ,  G06V40/16

CPC classification number: G06T13/80 ,  G06T7/174 ,  G06V40/167 ,  G06T2207/20084

Abstract: Provided are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes information concerning a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the target image and the information concerning the first face, two-dimensional (2D) deformations of the second face in the target image, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

公开(公告)号：US20240104789A1

公开(公告)日：2024-03-28

申请号：US17950945

申请日：2022-09-22

Applicant: Snap Inc.

Inventor： Arnab Ghosh ,  Jian Ren ,  Pavel Savchenkov ,  Sergey Tulyakov

IPC: G06T11/00 ,  G06F40/289 ,  G06F40/35 ,  G06V40/16

CPC classification number: G06T11/00 ,  G06F40/289 ,  G06F40/35 ,  G06V40/161

Abstract: A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The coordinates of a face in the image are determined, and the face of the user or another person is added to the image at the location. The final image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

公开(公告)号：US20240062008A1

公开(公告)日：2024-02-22

申请号：US17820437

申请日：2022-08-17

Applicant: Snap Inc.

Inventor： Arnab Ghosh ,  Jian Ren ,  Pavel Savchenkov ,  Sergey Tulyakov

IPC: G06F40/289 ,  G06T11/60 ,  H04L51/10 ,  G06F3/04842 ,  G06F16/583

CPC classification number: G06F40/289 ,  G06T11/60 ,  H04L51/10 ,  G06F3/04842 ,  G06F16/5846 ,  G06T2200/24

Abstract: A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The generated image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

公开(公告)号：US11741940B2

公开(公告)日：2023-08-29

申请号：US17355834

申请日：2021-06-23

Applicant: Snap Inc.

Inventor： Pavel Savchenkov ,  Maxim Lukin ,  Aleksandr Mashrabov

IPC: G10L13/00 ,  G10L13/08 ,  G06T13/40 ,  G06V40/16 ,  G06V10/764 ,  G06V10/82

CPC classification number: G10L13/00 ,  G06T13/40 ,  G06V10/764 ,  G06V10/82 ,  G06V40/171 ,  G10L13/08

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.

公开(公告)号：US20220284654A1

公开(公告)日：2022-09-08

申请号：US17751796

申请日：2022-05-24

Applicant: Snap Inc.

Inventor： Eugene Krokhalev ,  Aleksandr Mashrabov ,  Pavel Savchenkov

IPC: G06T13/80 ,  G06T7/174 ,  G06V40/16

Abstract: Disclosed are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes at least one input frame and the at least one input frame includes a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face in the target image, where the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression of the first face, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.