公开(公告)号：US11756250B2

公开(公告)日：2023-09-12

申请号：US17669270

申请日：2022-02-10

Applicant: Meta Platforms Technologies, LLC

Inventor： Alexander Richard ,  Michael Zollhoefer ,  Fernando De la Torre ,  Yaser Sheikh

IPC: G06T19/00 ,  G06T13/20 ,  G06T13/40 ,  G06T17/20 ,  G10L21/14 ,  G10L21/10

CPC classification number: G06T13/205 ,  G06T13/40 ,  G06T17/20 ,  G06T19/006 ,  G10L21/14 ,  G10L2021/105

Abstract: A method for training a three-dimensional model face animation model from speech, is provided. The method includes determining a first correlation value for a facial feature based on an audio waveform from a first subject, generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value, updating the first correlation value when a difference between the first mesh and a ground truth image of the first subject is greater than a pre-selected threshold, and providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first subject. A non-transitory, computer-readable medium storing instructions to cause a system to perform the above method, and the system, are also provided.

公开(公告)号：US11715248B2

公开(公告)日：2023-08-01

申请号：US17580486

申请日：2022-01-20

Applicant: Meta Platforms Technologies, LLC

Inventor： Jason Saragih ,  Stephen Anthony Lombardi ,  Shunsuke Saito ,  Tomas Simon Kreuz ,  Shih-En Wei ,  Kevyn Alex Anthony McPhail ,  Yaser Sheikh ,  Sai Bi

IPC: G06T13/40 ,  G06T15/04 ,  G06T15/60 ,  G06T15/50 ,  G06T11/00

CPC classification number: G06T13/40 ,  G06T11/001 ,  G06T15/04 ,  G06T15/506 ,  G06T15/60 ,  G06T2215/12

Abstract: A method for providing a relightable avatar of a subject to a virtual reality application is provided. The method includes retrieving multiple images including multiple views of a subject and generating an expression-dependent texture map and a view-dependent texture map for the subject, based on the images. The method also includes generating, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application, and providing the view of the subject to an immersive reality application running in a client device. A non-transitory, computer-readable medium storing instructions and a system that executes the instructions to perform the above method are also provided.

公开(公告)号：US20240303951A1

公开(公告)日：2024-09-12

申请号：US18636687

申请日：2024-04-16

Applicant: Meta Platforms Technologies, LLC

Inventor： Stephen Anthony Lombardi ,  Tomas Simon Kreuz ,  Jason Saragih ,  Gabriel Bailowitz Schwartz ,  Michael Zollhoefer ,  Yaser Sheikh

IPC: G06T19/20 ,  G06T15/06 ,  G06T17/20

CPC classification number: G06T19/20 ,  G06T15/06 ,  G06T17/20 ,  G06T2219/2012

Abstract: A method for training a real-time, modeling for animating an avatar for a subject is provided. The method includes collecting multiple images of a subject. The method also includes selecting a plurality of vertex positions in a guide mesh, indicative of a volumetric primitive enveloping the subject, determining a geometric attribute for the volumetric primitive including a position, a rotation, and a scale factor of the volumetric primitive, determining a payload attribute for each of the volumetric primitive, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the volumetric primitive, determining a loss factor for each point in the volumetric primitive based on the geometric attribute, the payload attribute and a ground truth value, and updating a three-dimensional model for the subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.