公开(公告)号：US20250095630A1

公开(公告)日：2025-03-20

申请号：US18966088

申请日：2024-12-02

Applicant: Google LLC

Inventor： Ye Jia ,  Zhifeng Chen ,  Yonghui Wu ,  Jonathan Shen ,  Ruoming Pang ,  Ron J. Weiss ,  Ignacio Lopez Moreno ,  Fei Ren ,  Yu Zhang ,  Quan Wang ,  Patrick An Phu Nguyen

IPC: G10L13/04 ,  G06N3/08 ,  G10L13/02 ,  G10L17/04 ,  G10L19/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech synthesis. The methods, systems, and apparatus include actions of obtaining an audio representation of speech of a target speaker, obtaining input text for which speech is to be synthesized in a voice of the target speaker, generating a speaker vector by providing the audio representation to a speaker encoder engine that is trained to distinguish speakers from one another, generating an audio representation of the input text spoken in the voice of the target speaker by providing the input text and the speaker vector to a spectrogram generation engine that is trained using voices of reference speakers to generate audio representations, and providing the audio representation of the input text spoken in the voice of the target speaker for output.

公开(公告)号：US11848002B2

公开(公告)日：2023-12-19

申请号：US17813361

申请日：2022-07-19

Applicant: Google LLC

Inventor： Ye Jia ,  Zhifeng Chen ,  Yonghui Wu ,  Jonathan Shen ,  Ruoming Pang ,  Ron J. Weiss ,  Ignacio Lopez Moreno ,  Fei Ren ,  Yu Zhang ,  Quan Wang ,  Patrick An Phu Nguyen

IPC: G10L13/04 ,  G10L17/04 ,  G10L19/00 ,  G06N3/08 ,  G10L13/02

CPC classification number: G10L13/04 ,  G10L17/04 ,  G10L19/00 ,  G06N3/08 ,  G10L2013/021

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech synthesis. The methods, systems, and apparatus include actions of obtaining an audio representation of speech of a target speaker, obtaining input text for which speech is to be synthesized in a voice of the target speaker, generating a speaker vector by providing the audio representation to a speaker encoder engine that is trained to distinguish speakers from one another, generating an audio representation of the input text spoken in the voice of the target speaker by providing the input text and the speaker vector to a spectrogram generation engine that is trained using voices of reference speakers to generate audio representations, and providing the audio representation of the input text spoken in the voice of the target speaker for output.

公开(公告)号：US20220068256A1

公开(公告)日：2022-03-03

申请号：US17005974

申请日：2020-08-28

Applicant: Google LLC

Inventor： Ye Jia ,  Byungha Chun ,  Yusuke Oda ,  Norman Casagrande ,  Tejas Iyer ,  Fan Luo ,  Russell John Wyatt Skerry-Ryan ,  Jonathan Shen ,  Yonghui Wu ,  Yu Zhang

IPC: G10L13/04 ,  G10L13/033 ,  G10L13/08 ,  G10L15/06

Abstract: A method of building a text-to-speech (TTS) system from a small amount of speech data includes receiving a first plurality of recorded speech samples from an assortment of speakers and a second plurality of recorded speech samples from a target speaker where the assortment of speakers does not include the target speaker. The method further includes training a TTS model using the first plurality of recorded speech samples from the assortment of speakers. Here, the trained TTS model is configured to output synthetic speech as an audible representation of a text input. The method also includes re-training the trained TTS model using the second plurality of recorded speech samples from the target speaker combined with the first plurality of recorded speech samples from the assortment of speakers. Here, the re-trained TTS model is configured to output synthetic speech resembling speaking characteristics of the target speaker.

公开(公告)号：US20240273311A1

公开(公告)日：2024-08-15

申请号：US18626745

申请日：2024-04-04

Applicant: Google LLC

Inventor： Ye Jia ,  Michelle Tadmor Ramanovich ,  Tal Remez ,  Roi Pomerantz

IPC: G06F40/58 ,  G10L13/02 ,  G10L13/10 ,  G10L19/16

CPC classification number: G06F40/58 ,  G10L13/02 ,  G10L13/10 ,  G10L19/16

Abstract: A direct speech-to-speech translation (S2ST) model includes an encoder configured to receive an input speech representation that to an utterance spoken by a source speaker in a first language and encode the input speech representation into a hidden feature representation. The S2ST model also includes an attention module configured to generate a context vector that attends to the hidden representation encoded by the encoder. The S2ST model also includes a decoder configured to receive the context vector generated by the attention module and predict a phoneme representation that corresponds to a translation of the utterance in a second different language. The S2ST model also includes a synthesizer configured to receive the context vector and the phoneme representation and generate a translated synthesized speech representation that corresponds to a translation of the utterance spoken in the different second language.

公开(公告)号：US12032920B2

公开(公告)日：2024-07-09

申请号：US17056554

申请日：2020-03-07

Applicant: Google LLC

Inventor： Ye Jia ,  Zhifeng Chen ,  Yonghui Wu ,  Melvin Johnson ,  Fadi Biadsy ,  Ron Weiss ,  Wolfgang Macherey

IPC: G06F40/47 ,  G06F40/58

CPC classification number: G06F40/47 ,  G06F40/58

Abstract: The present disclosure provides systems and methods that train and use machine-learned models such as, for example, sequence-to-sequence models, to perform direct and text-free speech-to-speech translation. In particular, aspects of the present disclosure provide an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation.

公开(公告)号：US11960852B2

公开(公告)日：2024-04-16

申请号：US17644351

申请日：2021-12-15

Applicant: Google LLC

Inventor： Ye Jia ,  Michelle Tadmor Ramanovich ,  Tal Remez ,  Roi Pomerantz

IPC: G06F40/58 ,  G10L13/02 ,  G10L13/10 ,  G10L19/16

CPC classification number: G06F40/58 ,  G10L13/02 ,  G10L13/10 ,  G10L19/16

Abstract: A direct speech-to-speech translation (S2ST) model includes an encoder configured to receive an input speech representation that to an utterance spoken by a source speaker in a first language and encode the input speech representation into a hidden feature representation. The S2ST model also includes an attention module configured to generate a context vector that attends to the hidden representation encoded by the encoder. The S2ST model also includes a decoder configured to receive the context vector generated by the attention module and predict a phoneme representation that corresponds to a translation of the utterance in a second different language. The S2ST model also includes a synthesizer configured to receive the context vector and the phoneme representation and generate a translated synthesized speech representation that corresponds to a translation of the utterance spoken in the different second language.

公开(公告)号：US20240112667A1

公开(公告)日：2024-04-04

申请号：US18525475

申请日：2023-11-30

Applicant: Google LLC

Inventor： Ye Jia ,  Zhifeng Chen ,  Yonghui Wu ,  Jonathan Shen ,  Ruoming Pang ,  Ron J. Weiss ,  Ignacio Lopez Moreno ,  Fei Ren ,  Yu Zhang ,  Quan Wang ,  Patrick An Phu Nguyen

IPC: G10L13/04 ,  G10L17/04 ,  G10L19/00

CPC classification number: G10L13/04 ,  G10L17/04 ,  G10L19/00 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech synthesis. The methods, systems, and apparatus include actions of obtaining an audio representation of speech of a target speaker, obtaining input text for which speech is to be synthesized in a voice of the target speaker, generating a speaker vector by providing the audio representation to a speaker encoder engine that is trained to distinguish speakers from one another, generating an audio representation of the input text spoken in the voice of the target speaker by providing the input text and the speaker vector to a spectrogram generation engine that is trained using voices of reference speakers to generate audio representations, and providing the audio representation of the input text spoken in the voice of the target speaker for output.

公开(公告)号：US11908448B2

公开(公告)日：2024-02-20

申请号：US17327076

申请日：2021-05-21

Applicant: Google LLC

Inventor： Isaac Elias ,  Jonathan Shen ,  Yu Zhang ,  Ye Jia ,  Ron J. Weiss ,  Yonghui Wu ,  Byungha Chun

IPC: G10L13/08 ,  G10L13/047 ,  G06F40/126 ,  G10L21/10 ,  G06N3/08 ,  G06N3/088 ,  G06N3/044 ,  G06N3/045 ,  G06N3/048

CPC classification number: G10L13/08 ,  G06F40/126 ,  G06N3/044 ,  G06N3/045 ,  G06N3/08 ,  G06N3/088 ,  G10L13/047 ,  G10L21/10 ,  G06N3/048

Abstract: A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence. The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.

公开(公告)号：US20230013777A1

公开(公告)日：2023-01-19

申请号：US17644351

申请日：2021-12-15

Applicant: Google LLC

Inventor： Ye Jia ,  Michelle Tadmor Ramanovich ,  Tal Remez ,  Roi Pomerantz

IPC: G06F40/58 ,  G10L13/02 ,  G10L13/10 ,  G10L19/16

Abstract: A direct speech-to-speech translation (S2ST) model includes an encoder configured to receive an input speech representation that to an utterance spoken by a source speaker in a first language and encode the input speech representation into a hidden feature representation. The S2ST model also includes an attention module configured to generate a context vector that attends to the hidden representation encoded by the encoder. The S2ST model also includes a decoder configured to receive the context vector generated by the attention module and predict a phoneme representation that corresponds to a translation of the utterance in a second different language. The S2ST model also includes a synthesizer configured to receive the context vector and the phoneme representation and generate a translated synthesized speech representation that corresponds to a translation of the utterance spoken in the different second language.

公开(公告)号：US20220301543A1

公开(公告)日：2022-09-22

申请号：US17326542

申请日：2021-05-21

Applicant: Google LLC

Inventor： Isaac Elias ,  Byungha Chun ,  Jonathan Shen ,  Ye Jia ,  Yu Zhang ,  Yonghui Wu

IPC: G10L13/08 ,  G10L13/04

Abstract: A method for training a non-autoregressive TTS model includes obtaining a sequence representation of an encoded text sequence concatenated with a variational embedding. The method also includes using a duration model network to predict a phoneme duration for each phoneme represented by the encoded text sequence. Based on the predicted phoneme durations, the method also includes learning an interval representation and an auxiliary attention context representation. The method also includes upsampling, using the interval representation and the auxiliary attention context representation, the sequence representation into an upsampled output specifying a number of frames. The method also includes generating, based on the upsampled output, one or more predicted mel-frequency spectrogram sequences for the encoded text sequence. The method also includes determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence and training the TTS model based on the final spectrogram loss.