公开(公告)号：US11080589B2

公开(公告)日：2021-08-03

申请号：US16504924

申请日：2019-07-08

Applicant: Google LLC

Inventor： Ron J. Weiss ,  Thang Minh Luong ,  Peter J. Liu ,  Colin Abraham Raffel ,  Douglas Eck

IPC: G06N3/04 ,  G06F17/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence including a respective output at each of multiple output time steps from respective encoded representations of inputs in an input sequence. The method includes, for each output time step, starting from the position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, traversing the encoded representations until an encoded representation is selected as a current context vector at the output time step. A decoder neural network processes the current context vector and a preceding output at the preceding output time step to generate a respective output score for each possible output and to update the hidden state of the decoder recurrent neural network. An output is selected for the output time step using the output scores.

公开(公告)号：US11210475B2

公开(公告)日：2021-12-28

申请号：US16518518

申请日：2019-07-22

Applicant: Google LLC

Inventor： Chung-Cheng Chiu ,  Colin Abraham Raffel

IPC: G06F17/00 ,  G06F40/40 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for enhanced attention mechanisms. In some implementations, data indicating an input sequence is received. The data is processed using an encoder neural network to generate a sequence of encodings. A series of attention outputs is determined using one or more attender modules. Determining each attention output can include (i) selecting an encoding from the sequence of encodings and (ii) determining attention over a proper subset of the sequence of encodings, where the proper subset of encodings is determined based on a position of the selected encoding in the sequence of encodings. The selections of encodings are also monotonic through the sequence of encodings. An output sequence is generated by processing the attention outputs using a decoder neural network. An output is provided that indicates a language sequence determined from the output sequence.

公开(公告)号：US12175202B2

公开(公告)日：2024-12-24

申请号：US17456958

申请日：2021-11-30

Applicant: Google LLC

Inventor： Chung-Cheng Chiu ,  Colin Abraham Raffel

IPC: G10L25/30 ,  G06F40/40 ,  G06N3/044 ,  G06N3/045

Abstract: A method includes receiving a sequence of audio features characterizing an utterance and processing, using an encoder neural network, the sequence of audio features to generate a sequence of encodings. At each of a plurality of output steps, the method also includes determining a corresponding hard monotonic attention output to select an encoding from the sequence of encodings, identifying a proper subset of the sequence of encodings based on a position of the selected encoding in the sequence of encodings, and performing soft attention over the proper subset of the sequence of encodings to generate a context vector at the corresponding output step. The method also includes processing, using a decoder neural network, the context vector generated at the corresponding output step to predict a probability distribution over possible output labels at the corresponding output step.

公开(公告)号：US20220083743A1

公开(公告)日：2022-03-17

申请号：US17456958

申请日：2021-11-30

Applicant: Google LLC

Inventor： Chung-Cheng Chiu ,  Colin Abraham Raffel

IPC: G06F40/40 ,  G06N3/04

Abstract: A method includes receiving a sequence of audio features characterizing an utterance and processing, using an encoder neural network, the sequence of audio features to generate a sequence of encodings. At each of a plurality of output steps, the method also includes determining a corresponding hard monotonic attention output to select an encoding from the sequence of encodings, identifying a proper subset of the sequence of encodings based on a position of the selected encoding in the sequence of encodings, and performing soft attention over the proper subset of the sequence of encodings to generate a context vector at the corresponding output step. The method also includes processing, using a decoder neural network, the context vector generated at the corresponding output step to predict a probability distribution over possible output labels at the corresponding output step.

公开(公告)号：US11354574B2

公开(公告)日：2022-06-07

申请号：US16859789

申请日：2020-04-27

Applicant: Google LLC

Inventor： Aurko Roy ,  Ian Goodfellow ,  Jacob Buckman ,  Colin Abraham Raffel

IPC: G06N3/08 ,  G06F17/16 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for increasing the security of neural network by discretizing neural network inputs. One of the methods includes receiving a network input for a neural network; processing the network input using a discretization layer, wherein the discretization layer is configured to generate a discretized network input comprising a respective discretized vector for each of the numeric values in the network input; and processing the discretized network input using the plurality of additional neural network layers to generate a network output for the network input.

公开(公告)号：US20190332919A1

公开(公告)日：2019-10-31

申请号：US16504924

申请日：2019-07-08

Applicant: Google LLC

Inventor： Ron J. Weiss ,  Thang Minh Luong ,  Peter J. Liu ,  Colin Abraham Raffel ,  Douglas Eck

IPC: G06N3/04 ,  G06F17/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence including a respective output at each of multiple output time steps from respective encoded representations of inputs in an input sequence. The method includes, for each output time step, starting from the position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, traversing the encoded representations until an encoded representation is selected as a current context vector at the output time step. A decoder neural network processes the current context vector and a preceding output at the preceding output time step to generate a respective output score for each possible output and to update the hidden state of the decoder recurrent neural network. An output is selected for the output time step using the output scores.

公开(公告)号：US20200257978A1

公开(公告)日：2020-08-13

申请号：US16859789

申请日：2020-04-27

Applicant: Google LLC

Inventor： Aurko Roy ,  Ian Goodfellow ,  Jacob Buckman ,  Colin Abraham Raffel

IPC: G06N3/08 ,  G06N3/04 ,  G06F17/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for increasing the security of neural network by discretizing neural network inputs. One of the methods includes receiving a network input for a neural network; processing the network input using a discretization layer, wherein the discretization layer is configured to generate a discretized network input comprising a respective discretized vector for each of the numeric values in the network input; and processing the discretized network input using the plurality of additional neural network layers to generate a network output for the network input.

公开(公告)号：US20200026760A1

公开(公告)日：2020-01-23

申请号：US16518518

申请日：2019-07-22

Applicant: Google LLC

Inventor： Chung-Cheng Chiu ,  Colin Abraham Raffel

IPC: G06F17/28 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for enhanced attention mechanisms. In some implementations, data indicating an input sequence is received. The data is processed using an encoder neural network to generate a sequence of encodings. A series of attention outputs is determined using one or more attender modules. Determining each attention output can include (i) selecting an encoding from the sequence of encodings and (ii) determining attention over a proper subset of the sequence of encodings, where the proper subset of encodings is determined based on a position of the selected encoding in the sequence of encodings. The selections of encodings are also monotonic through the sequence of encodings. An output sequence is generated by processing the attention outputs using a decoder neural network. An output is provided that indicates a language sequence determined from the output sequence.