公开(公告)号：US20240428493A1

公开(公告)日：2024-12-26

申请号：US18736552

申请日：2024-06-07

Applicant: UBTECH ROBOTICS CORP LTD

Inventor： WAN DING ,  Dongyan Huang ,  Xianjie Yang ,  Zehong Zheng ,  Penghul Li

IPC: G06T13/40 ,  G06T7/73 ,  G06V10/44 ,  G06V40/16 ,  G06V40/20 ,  G10L15/02

Abstract: A method for synthesizing a talking head video includes: obtaining speech data to be synthesized and observation data, wherein the observation data is data obtained through observation other than the speech data; performing feature extraction on the speech data to obtain speech features corresponding to the speech data, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data; performing temporal modeling on the speech features and first non-speech features to obtain low-dimensional representations, wherein the first non-speech features are non-speech features that are sensitive to temporal changes; and performing video synthesis based on the low-dimensional representations and second non-speech features, wherein the second non-speech features are non-speech features insensitive to temporal changes.