公开(公告)号：US20220253680A1

公开(公告)日：2022-08-11

申请号：US17665279

申请日：2022-02-04

Applicant: Google LLC

Inventor： Zhe Zhao ,  Maheswaran Sathiamoorthy ,  Lichan Hong ,  Yihua Chen ,  Ed Huai-hsin Chi ,  Aakanksha Chowdhery ,  Hussein Hazimeh

IPC: G06N3/04 ,  G06K9/62

Abstract: A system including a main neural network for performing one or more machine learning tasks on a network input to generate one or more network outputs. The main neural network includes a Mixture of Experts (MoE) subnetwork that includes a plurality of expert neural networks and a gating subsystem. The gating subsystem is configured to: apply a softmax function to a set of gating parameters having learned values to generate a respective softmax score for each of one or more of the plurality of expert neural networks; determine a respective weight for each of the one or more of the plurality of expert neural networks; select a proper subset of the plurality of expert neural networks; and combine the respective expert outputs generated by the one or more expert neural networks in the proper subset to generate one or more MoE outputs.