公开(公告)号：US12033259B2

公开(公告)日：2024-07-09

申请号：US17796399

申请日：2021-01-29

Applicant: Google LLC

Inventor： Vivek Kwatra ,  Christian Frueh ,  Avisek Lahiri ,  John Lewis

IPC: G06T13/20 ,  G06T13/40 ,  G06T17/20

CPC classification number: G06T13/205 ,  G06T13/40 ,  G06T17/20

Abstract: Provided is a framework for generating photorealistic 3D talking faces conditioned only on audio input. In addition, the present disclosure provides associated methods to insert generated faces into existing videos or virtual environments. We decompose faces from video into a normalized space that decouples 3D geometry, head pose, and texture. This allows separating the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. To stabilize temporal dynamics, we propose an auto-regressive approach that conditions the model on its previous visual state. We also capture face illumination in our model using audio-independent 3D texture normalization.

公开(公告)号：US20230343010A1

公开(公告)日：2023-10-26

申请号：US17796399

申请日：2021-01-29

Applicant: Google LLC

Inventor： Vivek Kwatra ,  Christian Frueh ,  Avisek Lahiri ,  John Lewis

IPC: G06T13/20 ,  G06T13/40 ,  G06T17/20

CPC classification number: G06T13/205 ,  G06T13/40 ,  G06T17/20

Abstract: Provided is a framework for generating photorealistic 3D talking faces conditioned only on audio input. In addition, the present disclosure provides associated methods to insert generated faces into existing videos or virtual environments. We decompose faces from video into a normalized space that decouples 3D geometry, head pose, and texture. This allows separating the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. To stabilize temporal dynamics, we propose an auto-regressive approach that conditions the model on its previous visual state. We also capture face illumination in our model using audio-independent 3D texture normalization.

公开(公告)号：US20240320892A1

公开(公告)日：2024-09-26

申请号：US18734327

申请日：2024-06-05

Applicant: Google LLC

Inventor： Vivek Kwatra ,  Christian Frueh ,  Avisek Lahiri ,  John Lewis

IPC: G06T13/20 ,  G06T13/40 ,  G06T17/20

CPC classification number: G06T13/205 ,  G06T13/40 ,  G06T17/20

Abstract: Provided is a framework for generating photorealistic 3D talking faces conditioned only on audio input. In addition, the present disclosure provides associated methods to insert generated faces into existing videos or virtual environments. We decompose faces from video into a normalized space that decouples 3D geometry, head pose, and texture. This allows separating the prediction problem into regressions over the 3D face shape and the corresponding 2D texture atlas. To stabilize temporal dynamics, we propose an auto-regressive approach that conditions the model on its previous visual state. We also capture face illumination in our model using audio-independent 3D texture normalization.