公开(公告)号：US20210374345A1

公开(公告)日：2021-12-02

申请号：US17336093

申请日：2021-06-01

Applicant: Google LLC

Inventor： Karthik Raman ,  Liu Yang ,  Mike Bendersky ,  Jiecao Chen ,  Marc Alexander Najork

IPC: G06F40/284 ,  G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a tuple of respective input sequences to generate an output. In one aspect, one of the systems includes a neural network comprising a plurality of encoder neural networks and a head neural network, each encoder neural network configured to: receive a respective input sequence from the tuple; process the respective input sequence using one or more encoder network layers to generate an encoded representation comprising a sequence of tokens; and process each of some or all of the tokens in the sequence of tokens using a projection layer to generate a lower-dimensional representation, and the head neural network configured to: receive lower-dimensional representations of a respective proper subset of the sequence of tokens generated by the encoder neural network; and process the lower-dimensional representations to generate the output.

公开(公告)号：US20210019654A1

公开(公告)日：2021-01-21

申请号：US16931862

申请日：2020-07-17

Applicant: Google LLC

Inventor： Xinnan Yu ,  Ankit Singh Rawat ,  Jiecao Chen ,  Ananda Theertha Suresh ,  Sanjiv Kumar

IPC: G06N20/00 ,  G06F40/20 ,  G06K9/62 ,  G06F17/14 ,  G06F17/18

Abstract: Systems and methods for low bias negative sampling of classes according to the sampled softmax method are described herein. The systems and methods can include training a machine-learned model for classifying inputs into one or more classes of a plurality of classes, each of the plurality of classes having an associated class embedding in a plurality of class embeddings. The systems and methods can include selecting, by the one or more computing devices, one or more negative classes from the plurality of classes based at least in part on a probability distribution approximating a softmax distribution, wherein the probability distribution is determined based at least in part on a Random Fourier Features map.

公开(公告)号：US12205005B2

公开(公告)日：2025-01-21

申请号：US16931862

申请日：2020-07-17

Applicant: Google LLC

Inventor： Xinnan Yu ,  Ankit Singh Rawat ,  Jiecao Chen ,  Ananda Theertha Suresh ,  Sanjiv Kumar

IPC: G06N3/08 ,  G06F17/14 ,  G06F17/18 ,  G06F18/2431 ,  G06F40/20 ,  G06N3/084 ,  G06N20/00 ,  G06N20/10 ,  G06V10/77

Abstract: Systems and methods for low bias negative sampling of classes according to the sampled softmax method are described herein. The systems and methods can include training a machine-learned model for classifying inputs into one or more classes of a plurality of classes, each of the plurality of classes having an associated class embedding in a plurality of class embeddings. The systems and methods can include selecting, by the one or more computing devices, one or more negative classes from the plurality of classes based at least in part on a probability distribution approximating a softmax distribution, wherein the probability distribution is determined based at least in part on a Random Fourier Features map.

公开(公告)号：US12182509B2

公开(公告)日：2024-12-31

申请号：US17336093

申请日：2021-06-01

Applicant: Google LLC

Inventor： Karthik Raman ,  Liu Yang ,  Mike Bendersky ,  Jiecao Chen ,  Marc Alexander Najork

IPC: G06F40/284 ,  G06N3/04 ,  G06N3/045 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a tuple of respective input sequences to generate an output. In one aspect, one of the systems includes a neural network comprising a plurality of encoder neural networks and a head neural network, each encoder neural network configured to: receive a respective input sequence from the tuple; process the respective input sequence using one or more encoder network layers to generate an encoded representation comprising a sequence of tokens; and process each of some or all of the tokens in the sequence of tokens using a projection layer to generate a lower-dimensional representation, and the head neural network configured to: receive lower-dimensional representations of a respective proper subset of the sequence of tokens generated by the encoder neural network; and process the lower-dimensional representations to generate the output.