公开(公告)号：US11922932B2

公开(公告)日：2024-03-05

申请号：US18194586

申请日：2023-03-31

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Tara N. Sainath ,  Yonghui Wu ,  Patrick An Phu Nguyen ,  Zhifeng Chen ,  Chung-Cheng Chiu ,  Anjuli Patricia Kannan

IPC: G10L15/197 ,  G10L15/02 ,  G10L15/06 ,  G10L15/16 ,  G10L15/22

CPC classification number: G10L15/197 ,  G10L15/02 ,  G10L15/063 ,  G10L15/16 ,  G10L15/22 ,  G10L2015/025

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer-readable storage media, for speech recognition using attention-based sequence-to-sequence models. In some implementations, audio data indicating acoustic characteristics of an utterance is received. A sequence of feature vectors indicative of the acoustic characteristics of the utterance is generated. The sequence of feature vectors is processed using a speech recognition model that has been trained using a loss function that uses a set of speech recognition hypothesis samples, the speech recognition model including an encoder, an attention module, and a decoder. The encoder and decoder each include one or more recurrent neural network layers. A sequence of output vectors representing distributions over a predetermined set of linguistic units is obtained. A transcription for the utterance is obtained based on the sequence of output vectors. Data indicating the transcription of the utterance is provided.

公开(公告)号：US11741966B2

公开(公告)日：2023-08-29

申请号：US17964141

申请日：2022-10-12

Applicant: GOOGLE LLC

Inventor： Asaf Aharoni ,  Arun Narayanan ,  Nir Shabat ,  Parisa Haghani ,  Galen Tsai Chuang ,  Yaniv Leviathan ,  Neeraj Gaur ,  Pedro J. Moreno Mengibar ,  Rohit Prakash Prabhavalkar ,  Zhongdi Qu ,  Austin Severn Waters ,  Tomer Amiaz ,  Michiel A. U. Bacchiani

IPC: G10L15/26 ,  H04M3/428 ,  H04M1/663 ,  G10L15/32 ,  H04M3/51 ,  H04M1/02

CPC classification number: G10L15/26 ,  G10L15/32 ,  H04M1/02 ,  H04M1/663 ,  H04M3/4286 ,  H04M3/5191

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

公开(公告)号：US20220366897A1

公开(公告)日：2022-11-17

申请号：US17815049

申请日：2022-07-26

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Golan Pundak ,  Tara N. Sainath

IPC: G10L15/16 ,  G10L15/26

Abstract: A method includes receiving audio data encoding an utterance and obtaining a set of bias phrases corresponding to a context of the utterance. Each bias phrase includes one or more words. The method also includes processing, using a speech recognition model, acoustic features derived from the audio to generate an output from the speech recognition model. The speech recognition model includes a first encoder configured to receive the acoustic features, a bias encoder configured to receive data indicating the obtained set of bias phrases, a bias encoder, and a decoder configured to determine likelihoods of sequences of speech elements based on output of the first attention module and output of the bias attention module. The method also includes determining a transcript for the utterance based on the likelihoods of sequences of speech elements.

公开(公告)号：US11270687B2

公开(公告)日：2022-03-08

申请号：US16861190

申请日：2020-04-28

Applicant: Google LLC

Inventor： Ke Hu ,  Antoine Jean Bruguier ,  Tara N. Sainath ,  Rohit Prakash Prabhavalkar ,  Golan Pundak

IPC: G10L15/30 ,  G10L15/06 ,  G10L15/02 ,  G10L15/187 ,  G10L15/193 ,  G10L15/28 ,  G10L15/32 ,  G10L25/30

Abstract: A method includes receiving audio data encoding an utterance spoken by a native speaker of a first language, and receiving a biasing term list including one or more terms in a second language different than the first language. The method also includes processing, using a speech recognition model, acoustic features derived from the audio data to generate speech recognition scores for both wordpieces and corresponding phoneme sequences in the first language. The method also includes rescoring the speech recognition scores for the phoneme sequences based on the one or more terms in the biasing term list, and executing, using the speech recognition scores for the wordpieces and the rescored speech recognition scores for the phoneme sequences, a decoding graph to generate a transcription for the utterance.

公开(公告)号：US20220005465A1

公开(公告)日：2022-01-06

申请号：US17448119

申请日：2021-09-20

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Zhifeng Chen ,  Bo Li ,  Chung-cheng Chiu ,  Kanury Kanishka Rao ,  Yonghui Wu ,  Ron J. Weiss ,  Navdeep Jaitly ,  Michiel A.u. Bacchiani ,  Tara N. Sainath ,  Jan Kazimierz Chorowski ,  Anjuli Patricia Kannan ,  Ekaterina Gonina ,  Patrick An Phu Nguyen

IPC: G10L15/16 ,  G10L15/22 ,  G10L15/02 ,  G06N3/08 ,  G10L15/06 ,  G10L25/30

Abstract: A method for performing speech recognition using sequence-to-sequence models includes receiving audio data for an utterance and providing features indicative of acoustic characteristics of the utterance as input to an encoder. The method also includes processing an output of the encoder using an attender to generate a context vector, generating speech recognition scores using the context vector and a decoder trained using a training process, and generating a transcription for the utterance using word elements selected based on the speech recognition scores. The transcription is provided as an output of the ASR system.

公开(公告)号：US20210089916A1

公开(公告)日：2021-03-25

申请号：US17112966

申请日：2020-12-04

Applicant: Google LLC

Inventor： Ouais Alsharif ,  Rohit Prakash Prabhavalkar ,  Ian C. McGraw ,  Antoine Jean Bruguier

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a compressed recurrent neural network (RNN). One of the systems includes a compressed RNN, the compressed RNN comprising a plurality of recurrent layers, wherein each of the recurrent layers has a respective recurrent weight matrix and a respective inter-layer weight matrix, and wherein at least one of recurrent layers is compressed such that a respective recurrent weight matrix of the compressed layer is defined by a first compressed weight matrix and a projection matrix and a respective inter-layer weight matrix of the compressed layer is defined by a second compressed weight matrix and the projection matrix.

公开(公告)号：US12254883B2

公开(公告)日：2025-03-18

申请号：US18635974

申请日：2024-04-15

Applicant: GOOGLE LLC

Inventor： Asaf Aharoni ,  Arun Narayanan ,  Nir Shabat ,  Parisa Haghani ,  Galen Tsai Chuang ,  Yaniv Leviathan ,  Neeraj Gaur ,  Pedro J. Moreno Mengibar ,  Rohit Prakash Prabhavalkar ,  Zhongdi Qu ,  Austin Severn Waters ,  Tomer Amiaz ,  Michiel A. U. Bacchiani

IPC: G10L15/26 ,  G10L15/32 ,  H04M1/02 ,  H04M1/663 ,  H04M3/428 ,  H04M3/51

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for an automated calling system are disclosed. In one aspect, a method includes the actions of receiving audio data of an utterance spoken by a user who is having a telephone conversation with a bot. The actions further include determining a context of the telephone conversation. The actions further include determining a user intent of a first previous portion of the telephone conversation spoken by the user and a bot intent of a second previous portion of the telephone conversation outputted by a speech synthesizer of the bot. The actions further include, based on the audio data of the utterance, the context of the telephone conversation, the user intent, and the bot intent, generating synthesized speech of a reply by the bot to the utterance. The actions further include, providing, for output, the synthesized speech.

公开(公告)号：US20250078815A1

公开(公告)日：2025-03-06

申请号：US18826135

申请日：2024-09-05

Applicant: Google LLC

Inventor： Shaojin Ding ,  David Qiu ,  David Rim ,  Amir Yazdanbakhsh ,  Yanzhang He ,  Zhonglin Han ,  Rohit Prakash Prabhavalkar ,  Weiran Wang ,  Bo Li ,  Jian Li ,  Tara N. Sainath ,  Shivani Agrawal ,  Oleg Rybakov

IPC: G10L15/06

Abstract: A method includes obtaining a plurality of training samples that each include a respective speech utterance and a respective textual utterance representing a transcription of the respective speech utterance. The method also includes fine-tuning, using quantization and sparsity aware training with native integer operations, a pre-trained automatic speech recognition (ASR) model on the plurality of training samples. Here, the pre-trained ASR model includes a plurality of weights and the fine-tuning includes pruning one or more weights of the plurality of weights using a sparsity mask and quantizing each weight of the plurality of weights based on an integer with a fixed-bit width. The method also includes providing the fine-tuned ASR model to a user device.

公开(公告)号：US20240379095A1

公开(公告)日：2024-11-14

申请号：US18782001

申请日：2024-07-23

Applicant: Google LLC

Inventor： Rohit Prakash Prabhavalkar ,  Golan Pundak ,  Tara N. Sainath

IPC: G10L15/16 ,  G10L15/26

Abstract: A method includes receiving audio data encoding an utterance and obtaining a set of bias phrases corresponding to a context of the utterance. Each bias phrase includes one or more words. The method also includes processing, using a speech recognition model, acoustic features derived from the audio to generate an output from the speech recognition model. The speech recognition model includes a first encoder configured to receive the acoustic features, a bias encoder configured to receive data indicating the obtained set of bias phrases, a bias encoder, and a decoder configured to determine likelihoods of sequences of speech elements based on output of the first attention module and output of the bias attention module. The method also includes determining a transcript for the utterance based on the likelihoods of sequences of speech elements.

公开(公告)号：US20240304181A1

公开(公告)日：2024-09-12

申请号：US18598523

申请日：2024-03-07

Applicant: Google LLC

Inventor： Guru Prakash Arumugam ,  Shuo-yiin Chang ,  Shaan Jagdeep Patrick Bijwadia ,  Weiran Wang ,  Quan Wang ,  Rohit Prakash Prabhavalkar ,  Tara N. Sainath

IPC: G10L15/06

CPC classification number: G10L15/063

Abstract: A method includes receiving a plurality of training samples spanning multiple different domains. Each corresponding training sample includes audio data characterizing an utterance paired with a corresponding transcription of the utterance. The method also includes re-labeling each corresponding training sample of the plurality of training samples by annotating the corresponding transcription of the utterance with one or more speaker tags. Each speaker tag indicates a respective segment of the transcription for speech that was spoken by a particular type of speaker. The method also includes training a multi-domain speech recognition model on the re-labeled training samples to teach the multi-domain speech recognition model to learn to share parameters for recognizing speech across each of the different multiple different domains.