公开(公告)号：US20230077928A1

公开(公告)日：2023-03-16

申请号：US17474928

申请日：2021-09-14

Applicant: Google LLC

Inventor： James Patrick Lee-Thorp ,  Joshua Timothy Ainslie ,  Ilya Eckstein ,  Santiago Ontañón

IPC: G06N20/00 ,  G06F17/14

Abstract: Transformer systems and methods of using such transformer systems including computer programs encoded on a computer storage medium, for performing a deep learning task on an input sequence to generate an encoded output. In one aspect, one of the transformer systems includes an encoder architecture block, comprising: a spectral transform mixing layer that receives input embeddings of input tokens and generates, as output, a spectral transform output along a sequence dimension of the input embeddings; and a feed forward layer that receives an input based on the input embeddings of input tokens and the spectral transform output and generates an output for a subsequent processing block.