公开(公告)号：US20220108212A1

公开(公告)日：2022-04-07

申请号：US17308033

申请日：2021-05-04

Applicant: Apple Inc.

Inventor： Shuangfei ZHAI ,  Walter A. TALBOTT ,  Nitish SRIVASTAVA ,  Chen HUANG ,  Hanlin GOH ,  Joshua M. SUSSKIND

IPC: G06N20/00 ,  G06N5/04 ,  G06F17/16 ,  G06F40/58 ,  G06T3/40

Abstract: Attention-free transformers are disclosed. Various implementations of attention-free transformers include a gating and pooling operation that allows the attention-free transformers to provide comparable or better results to those of a standard attention-based transformer, with improved efficiency and reduced computational complexity with respect to space and time.