公开(公告)号：US12079927B2

公开(公告)日：2024-09-03

申请号：US17506248

申请日：2021-10-20

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T15/50 ,  G06N3/08 ,  G06V10/60

CPC classification number: G06T15/506 ,  G06N3/08 ,  G06V10/60

Abstract: A messaging system performs image processing to estimate lighting properties with neural networks for images provided by users of the messaging system. A method of estimating light properties includes receiving an input image with first lighting properties and processing the input image using a convolutional neural network to generate an estimate of the first lighting properties. The method may further include modifying the input image with an augmentation to generate a modified input image, where the augmentation has second lighting properties, and changing the second lighting properties of the augmentation in the modified input image to the estimate of the first lighting properties.

公开(公告)号：US10776981B1

公开(公告)日：2020-09-15

申请号：US16551756

申请日：2019-08-27

Applicant: SNAP INC.

Inventor： Sergei Gorbatyuk ,  Nikolai Smirnov ,  Aleksandr Mashrabov ,  Egor Nemchinov

IPC: G06T7/70 ,  G06T13/40 ,  G06T7/194 ,  G06F3/0482 ,  G06F3/0484 ,  G06T7/20 ,  G06T19/00

Abstract: Provided are systems and methods for animating a single image of a human body and applying effects. An example method includes providing, by a computer device, a database of motions; receiving, by a computing device, an input image, the input image including a body of a person; receiving, by the computing device, a user input including a motion selected from the database of motions; segmenting, by the computing device, the input image into a body portion and a background portion; generating, by the computing device and based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion; and displaying, by the computing device, the generated video; receiving, by a computer device, a further user input including clothes, scene, illumination effect, and additional objects; and, while generating the video, modifying frames of the video based on the further user input.

公开(公告)号：US11830129B2

公开(公告)日：2023-11-28

申请号：US17385462

申请日：2021-07-26

Applicant: Snap Inc.

Inventor： Yuriy Volkov ,  Egor Nemchinov ,  Gleb Dmukhin

IPC: G06T5/50 ,  G06T7/73 ,  G06T19/20 ,  G06T15/50

CPC classification number: G06T15/503 ,  G06T7/75 ,  G06T15/506 ,  G06T19/20 ,  G06T2200/24 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2219/2012

Abstract: A messaging system performs image processing to relight objects with neural networks for images provided by users of the messaging system. A method of relighting objects with neural networks includes receiving an input image with first lighting properties comprising an object with second lighting properties and processing the input image using a convolutional neural network to generate an output image with the first lighting properties and comprising the object with third lighting properties, where the convolutional neural network is trained to modify the second lighting properties to be consistent with lighting conditions indicated by the first lighting properties to generate the third lighting properties. The method further includes modifying the second lighting properties of the object to generate the object with modified second lighting properties and blending the third lighting properties with the modified second lighting properties to generate a modified output image comprising the object with fourth lighting properties.

公开(公告)号：US11727617B2

公开(公告)日：2023-08-15

申请号：US17695902

申请日：2022-03-16

Applicant: Snap Inc.

Inventor： Egor Nemchinov ,  Sergei Gorbatyuk ,  Aleksandr Mashrabov ,  Egor Spirin ,  Iaroslav Sokolov ,  Andrei Smirdin ,  Igor Tukh

IPC: G06T13/40 ,  G06T7/194 ,  G06N3/08 ,  G06T3/00 ,  G06T15/04 ,  G06T17/20

CPC classification number: G06T13/40 ,  G06N3/08 ,  G06T3/0037 ,  G06T7/194 ,  G06T15/04 ,  G06T17/20

Abstract: Disclosed are systems and methods for single image-based body animation. An example method includes receiving an input image, the input image including a body image of a person, extracting the body image of the person from the input image, fitting a generic model to the body image, where the generic model is configured to receive a set of pose parameters corresponding to a pose of the person and generate a generic body shape adopting the pose, generating a three-dimensional (3D) model, where the 3D model is configured to receive a set of further pose parameters corresponding to the pose of the person and generate an output image of the person adopting the pose, the output image including a feature of the body image being omitted from the generic body shape, and providing a further set of further pose parameters to generate a frame of an output video.

公开(公告)号：US11410365B2

公开(公告)日：2022-08-09

申请号：US17019875

申请日：2020-09-14

Applicant: Snap Inc.

Inventor： Sergei Gorbatyuk ,  Nikolai Smirnov ,  Aleksandr Mashrabov ,  Egor Nemchinov

IPC: G06T13/40 ,  G06T7/194 ,  G06T7/70 ,  G06F3/0482 ,  G06F3/04845 ,  G06T7/20 ,  G06T19/00

Abstract: Provided are systems and methods for animating a single image of a human body and applying effects. An example method includes providing a database of motions; receiving an input image including a body of a person; receiving a user input including a motion selected from the database of motions; segmenting the input image into a body portion and a background portion; fitting the body portion to a hair model; generating, based on the body portion and the selected motion, a video featuring the body of the person repeating the selected motion; where generating the video includes detecting positions of key points associated with a head of the person in a frame of the video, generating an image of hair of the person based on the positions of the key points and the hair model, and inserting the image of the hair in the frame; and displaying the generated video.

公开(公告)号：US20240371085A1

公开(公告)日：2024-11-07

申请号：US18770353

申请日：2024-07-11

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T15/50 ,  G06N3/08 ,  G06V10/60

Abstract: A messaging system performs image processing to estimate lighting properties with neural networks for images provided by users of the messaging system. A method of estimating light properties includes receiving an input image with first lighting properties and processing the input image using a convolutional neural network to generate an estimate of the first lighting properties. The method may further include modifying the input image with an augmentation to generate a modified input image, where the augmentation has second lighting properties, and changing the second lighting properties of the augmentation in the modified input image to the estimate of the first lighting properties.

公开(公告)号：US20240312099A1

公开(公告)日：2024-09-19

申请号：US18673574

申请日：2024-05-24

Applicant: Snap Inc.

Inventor： Sergei Gorbatyuk ,  Nikolai Smirnov ,  Aleksandr Mashrabov ,  Egor Nemchinov

IPC: G06T13/40 ,  G06F3/0482 ,  G06F3/04845 ,  G06T7/194 ,  G06T7/20 ,  G06T7/70 ,  G06T19/00

CPC classification number: G06T13/40 ,  G06F3/0482 ,  G06F3/04845 ,  G06T7/194 ,  G06T7/20 ,  G06T7/70 ,  G06T19/00 ,  G06T2200/24 ,  G06T2207/10016 ,  G06T2207/30196

Abstract: Provided are systems and methods for animating a single image of a human body and applying effects. An example method includes receiving an image of a body, receiving, via a user interface, a user input including parameters associated with a motion, fitting a portion of the image of the body to a hair model designed to generate a hair image corresponding to the motion, where the portion of the image of the body includes hair, and generating a video featuring the body repeating the motion, where the generation of the video is based on the image of the body, the parameters associated with the motion, and the hair image generated by the hair model.

公开(公告)号：US11308676B2

公开(公告)日：2022-04-19

申请号：US17062309

申请日：2020-10-02

Applicant: Snap Inc.

Inventor： Egor Nemchinov ,  Sergei Gorbatyuk ,  Aleksandr Mashrabov ,  Egor Spirin ,  Iaroslav Sokolov ,  Andrei Smirdin ,  Igor Tukh

IPC: G06T13/40 ,  G06T7/194 ,  G06N3/08 ,  G06T3/00 ,  G06T15/04 ,  G06T17/20

Abstract: Provided are systems and methods for single image-based body animation. An example method includes receiving an input image that includes a body of a person and segmenting the input image into a body portion and a background portion. The method further includes fitting a model to the body portion. The model is configured to receive a set of pose parameters representing a pose of the body and generate an output image including an image of the body adopting the pose. The method further includes receiving a series of further sets of pose parameters, each representing at least one of further poses of the body. The further sets of pose parameters are generated using a generic model. The method also includes generating a series of output images of the body adopting the further poses and generating an output video based on the series of output images.

公开(公告)号：US20210019929A1

公开(公告)日：2021-01-21

申请号：US17062309

申请日：2020-10-02

Applicant: Snap Inc.

Inventor： Egor Nemchinov ,  Sergei Gorbatyuk ,  Aleksandr Mashrabov ,  Egor Spirin ,  Iaroslav Sokolov ,  Andrei Smirdin ,  Igor Tukh

IPC: G06T13/40 ,  G06T7/194 ,  G06T15/04 ,  G06T3/00 ,  G06N3/08 ,  G06T17/20

Abstract: Provided are systems and methods for single image-based body animation. An example method includes receiving an input image that includes a body of a person and segmenting the input image into a body portion and a background portion. The method further includes fitting a model to the body portion. The model is configured to receive a set of pose parameters representing a pose of the body and generate an output image including an image of the body adopting the pose. The method further includes receiving a series of further sets of pose parameters, each representing at least one of further poses of the body. The further sets of pose parameters are generated using a generic model. The method also includes generating a series of output images of the body adopting the further poses and generating an output video based on the series of output images.

公开(公告)号：US12211166B2

公开(公告)日：2025-01-28

申请号：US18386515

申请日：2023-11-02

Applicant: Snap Inc.

Inventor： Gleb Dmukhin ,  Egor Nemchinov ,  Yurii Volkov

IPC: G06T19/20 ,  G06N3/02 ,  G06N3/047 ,  G06N3/0475 ,  G06N3/0895 ,  G06N3/094 ,  G06T17/20 ,  H04L51/10

Abstract: A messaging system processes three-dimensional (3D) models to generate ground truths for training machine learning models for applications of the messaging system. A method of generating ground truths for machine learning includes generating a plurality of first rendered images from a first 3D base model where each first rendered image includes the 3D base model modified by first augmentations of a plurality of augmentations. The method further includes determining for a second 3D base model incompatible augmentations of the first plurality of augmentations, where the incompatible augmentations indicate changes to fixed features of the second 3D base model, and generating a plurality of second rendered images from a second 3D base model, each second rendered image comprising the second 3D base model modified by second augmentations, the second augmentations corresponding to the first augmentations of a corresponding first rendered image, where the second augmentations comprises augmentations of the first augmentations that are not incompatible augmentations.