公开(公告)号：US11501171B2

公开(公告)日：2022-11-15

申请号：US17555535

申请日：2021-12-20

Applicant: ZHEJIANG LAB

Inventor： Hongsheng Wang ,  Enping Wang ,  Zailiang Yu

IPC: G06N3/08 ,  G06F40/40

Abstract: Disclosed are an automatic compression method and platform for a pre-trained language model based on multilevel knowledge distillation. The method includes the following steps: step 1, constructing multilevel knowledge distillation, and distilling a knowledge structure of a large model at three different levels: a self-attention unit, a hidden layer state and an embedded layer; step 2, training a knowledge distillation network of meta-learning to generate a general compression architecture of a plurality of pre-trained language models; and step 3, searching for an optimal compression structure based on an evolutionary algorithm. Firstly, the knowledge distillation based on meta-learning is studied to generate the general compression architecture of the plurality of pre-trained language models; and secondly, on the basis of a trained meta-learning network, the optimal compression structure is searched for via the evolutionary algorithm, so as to obtain an optimal general compression architecture of the pre-trained language model independent of tasks.