公开(公告)号：US20230004717A1

公开(公告)日：2023-01-05

申请号：US17572068

申请日：2022-01-10

Applicant: BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

Inventor： Lijie WANG ,  Shuai ZHANG ,  Xinyan XIAO ,  Yue CHANG ,  Tingting LI

IPC: G06F40/289 ,  G06N20/00

Abstract: The present disclosure provides a method and apparatus for acquiring a pre-trained model, an electronic device and a storage medium, and relates to the field of artificial intelligence, such as the natural language processing field, the deep learning field, or the like. The method may include: adding, in a process of training a pre-trained model using training sentences, a learning objective corresponding to syntactic information for a self-attention module in the pre-trained model; and training the pre-trained model according to the defined learning objective. The solution of the present disclosure may improve a performance of the pre-trained model, and reduce consumption of computing resources, or the like.