公开(公告)号：US11929060B2

公开(公告)日：2024-03-12

申请号：US17170836

申请日：2021-02-08

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Andrew Rosenberg ,  Bhuvana Ramabhadran ,  Pedro Jose Moreno Mengibar

IPC: G10L15/06 ,  G06N3/04 ,  G06N3/044 ,  G06N3/045 ,  G06N3/08 ,  G06N3/088 ,  G10L13/02 ,  G10L15/16 ,  G10L15/197

CPC classification number: G10L15/063 ,  G06N3/044 ,  G06N3/045 ,  G06N3/088 ,  G10L13/02 ,  G10L15/16 ,  G10L15/197 ,  G10L2015/0635

Abstract: A method for training a speech recognition model includes receiving a set of training utterance pairs each including a non-synthetic speech representation and a synthetic speech representation of a same corresponding utterance. At each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, the method also includes determining a consistent loss term for the corresponding training utterance pair based on a first probability distribution over possible non-synthetic speech recognition hypotheses generated for the corresponding non-synthetic speech representation and a second probability distribution over possible synthetic speech recognition hypotheses generated for the corresponding synthetic speech representation. The first and second probability distributions are generated for output by the speech recognition model. The method also includes updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair.

公开(公告)号：US20220068255A1

公开(公告)日：2022-03-03

申请号：US17454536

申请日：2021-11-11

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Andrew M. Rosenberg ,  Bhuvana Ramabhadran ,  Pedro J. Moreno Mengibar

IPC: G10L13/00 ,  G10L13/08 ,  G10L15/06

Abstract: A method for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in unison includes obtaining a plurality of training text utterances. At each of a plurality of output steps for each training text utterance, the method also includes generating, for output by the GAN-Based TTS model, a synthetic speech representation of the corresponding training text utterance, and determining, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

公开(公告)号：US11222620B2

公开(公告)日：2022-01-11

申请号：US16869552

申请日：2020-05-07

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Andrew M. Rosenberg ,  Bhuvana Ramabhadran ,  Pedro J. Moreno Mengibar

IPC: G10L15/26 ,  G10L13/00 ,  G10L13/08 ,  G10L15/06

Abstract: A method for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in unison includes obtaining a plurality of training text utterances. At each of a plurality of output steps for each training text utterance, the method also includes generating, for output by the GAN-Based TTS model, a synthetic speech representation of the corresponding training text utterance, and determining, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

公开(公告)号：US20210350786A1

公开(公告)日：2021-11-11

申请号：US16869552

申请日：2020-05-07

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Andrew M. Rosenberg ,  Bhuvana Ramabhadran ,  Pedro J. Moreno Mengibar

IPC: G10L13/04 ,  G10L15/06 ,  G10L13/08

Abstract: A method for training a generative adversarial network (GAN)-based text-to-speech (TTS) model and a speech recognition model in unison includes obtaining a plurality of training text utterances. At each of a plurality of output steps for each training text utterance, the method also includes generating, for output by the GAN-Based TTS model, a synthetic speech representation of the corresponding training text utterance, and determining, using an adversarial discriminator of the GAN, an adversarial loss term indicative of an amount of acoustic noise disparity in one of the non-synthetic speech representations selected from the set of spoken training utterances relative to the corresponding synthetic speech representation of the corresponding training text utterance. The method also includes updating parameters of the GAN-based TTS model based on the adversarial loss term determined at each of the plurality of output steps for each training text utterance of the plurality of training text utterances.

公开(公告)号：US12272348B2

公开(公告)日：2025-04-08

申请号：US17655030

申请日：2022-03-16

Applicant: Google LLC

Inventor： Bhuvana Ramabhadran ,  Zhehuai Chen ,  Fadi Biadsy ,  Pedro J. Moreno Mengibar

IPC: G10L13/027 ,  G10L13/047 ,  G10L15/16 ,  G10L15/22 ,  G10L25/18

Abstract: A method for speech conversion includes receiving, as input to an encoder of a speech conversion model, an input spectrogram corresponding to an utterance, the encoder including a stack of self-attention blocks. The method further includes generating, as output from the encoder, an encoded spectrogram and receiving, as input to a spectrogram decoder of the speech conversion model, the encoded spectrogram generated as output from the encoder. The method further includes generating, as output from the spectrogram decoder, an output spectrogram corresponding to a synthesized speech representation of the utterance.

公开(公告)号：US12159617B2

公开(公告)日：2024-12-03

申请号：US17808091

申请日：2022-06-21

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Bhuvana Ramabhadran ,  Andrew M. Rosenberg ,  Yu Zhang ,  Pedro J. Moreno Mengibar

IPC: G10L15/06 ,  G10L13/047 ,  G10L13/08 ,  G10L15/16

Abstract: A method includes receiving training data that includes unspoken text utterances and un-transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances and the un-transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.

公开(公告)号：US20240282292A1

公开(公告)日：2024-08-22

申请号：US18654278

申请日：2024-05-03

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Bhuvana Ramabhadran ,  Andrew Rosenberg ,  Yu Zhang ,  Pedro J. Moreno Mengibar

IPC: G10L13/047 ,  G10L13/08 ,  G10L13/10

CPC classification number: G10L13/047 ,  G10L13/086 ,  G10L13/10

Abstract: A method for training a speech recognition model includes obtaining a multilingual text-to-speech (TTS) model. The method also includes generating a native synthesized speech representation for an input text sequence in a first language that is conditioned on speaker characteristics of a native speaker of the first language. The method also includes generating a cross-lingual synthesized speech representation for the input text sequence in the first language that is conditioned on speaker characteristics of a native speaker of a different second language. The method also includes generating a first speech recognition result for the native synthesized speech representation and a second speech recognition result for the cross-lingual synthesized speech representation. The method also includes determining a consistent loss term based on the first speech recognition result and the second speech recognition result and updating parameters of the speech recognition model based on the consistent loss term.

公开(公告)号：US20240153484A1

公开(公告)日：2024-05-09

申请号：US18494324

申请日：2023-10-25

Applicant: Google LLC

Inventor： Andrew M. Rosenberg ,  Takaaki Saeki ,  Zhehuai Chen ,  Byungha Chun ,  Bhuvana Ramabhadran

IPC: G10L13/047 ,  G10L15/06 ,  G10L15/16

CPC classification number: G10L13/047 ,  G10L15/063 ,  G10L15/16

Abstract: A method includes receiving training data that includes a plurality of sets of text-to-speech (TTS) spoken utterances each associated with a respective language and including TTS utterances of synthetic speech spoken that includes a corresponding reference speech representation paired with a corresponding input text sequence. For each TTS utterance in each set of the TTS spoken training utterances of the received training data, the method includes generating a corresponding TTS encoded textual representation for the corresponding input text sequence, generating a corresponding speech encoding for the corresponding TTS utterance of synthetic speech, generating a shared encoder output, generating a predicted speech representation for the corresponding TTS utterance of synthetic speech, and determining a reconstruction loss. The method also includes training a TTS model based on the reconstruction losses determined for the TTS utterances in each set of the TTS spoken training utterances.

公开(公告)号：US20230017892A1

公开(公告)日：2023-01-19

申请号：US17808091

申请日：2022-06-21

Applicant: Google LLC

Inventor： Zhehuai Chen ,  Bhuvana Ramabhadran ,  Andrew M. Rosenberg ,  Yu Zhang ,  Pedro J. Moreno Mengibar

IPC: G10L13/047 ,  G10L13/08

Abstract: A method includes receiving training data that includes unspoken text utterances and un-transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances and the un-transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.

公开(公告)号：US20230013587A1

公开(公告)日：2023-01-19

申请号：US17722264

申请日：2022-04-15

Applicant: Google LLC

Inventor： Andrew Rosenberg ,  Zhehuai Chen ,  Bhuvana Ramabhadran ,  Pedro J. Moreno Mengibar ,  Gary Wang ,  Yu Zhang

IPC: G10L19/00 ,  G10L13/02 ,  G10L15/26

Abstract: A method includes receiving training data that includes unspoken text utterances, un-transcribed non-synthetic speech utterances, and transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances, the un-transcribed non-synthetic speech utterances, and the transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.