公开(公告)号：US20220350965A1

公开(公告)日：2022-11-03

申请号：US17864636

申请日：2022-07-14

Applicant: BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

Inventor： Tongyang LIU ,  Shu WANG ,  Wanli CHANG ,  Wei ZHENG ,  Zhifan FENG ,  Chunguang CHAI ,  Yong ZHU

IPC: G06F40/211 ,  G06F40/30 ,  G06F40/109 ,  G06N3/08

Abstract: A method for generating a pre-trained language model, includes: obtaining sample files; obtaining typography structure information and text information of the sample files by parsing the sample files; obtaining a plurality of task models of a pre-trained language model; obtaining a trained pre-trained language model by jointly training the pre-trained language model and the plurality of task models according to the typography structure information and the text information; and generating a target pre-trained language model by fine-tuning the trained pre-trained language model according to the typography structure information and the text information.