公开(公告)号：US20230229912A1

公开(公告)日：2023-07-20

申请号：US18123768

申请日：2023-03-20

Applicant: Huawei Technologies Co., Ltd.

Inventor： Wei ZHANG ,  Lu HOU ,  Yichun YIN ,  Lifeng SHANG

IPC: G06N3/08

CPC classification number: G06N3/08

Abstract: A model compression method is provided, which can be applied to the field of artificial intelligence. The method includes: obtaining a first neural network model, a second neural network model, and a third neural network model; processing first to-be-processed data using the first neural network model, to obtain a first output; processing the first to-be-processed data using the third neural network model, to obtain a second output; determining a first target loss based on the first output and the second output, and updating the second neural network model based on the first target loss, to obtain an updated second neural network model; and compressing the updated second neural network model to obtain a target neural network model. The model generated based on the method has higher processing precision.

公开(公告)号：US20240152770A1

公开(公告)日：2024-05-09

申请号：US18411616

申请日：2024-01-12

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Hang XU ,  Xiaozhe REN ,  Yichun YIN ,  Li QIAN ,  Zhenguo LI ,  Xin JIANG ,  Jiahui GAO

IPC: G06N3/0985 ,  G06N3/04

CPC classification number: G06N3/0985 ,  G06N3/04

Abstract: This application relates to the artificial intelligence field, and discloses a neural network search method and a related apparatus. The neural network search method includes: constructing attention heads in transformer layers by sampling a plurality of candidate operators during model search, to construct a plurality of candidate neural networks, and comparing performance of the plurality of candidate neural networks to select a target neural network with higher performance. In this application, a transformer model is constructed with reference to model search, so that a new attention structure with better performance than an original self-attention mechanism can be generated, and effect in a wide range of downstream tasks is significantly improved.

公开(公告)号：US20230274144A1

公开(公告)日：2023-08-31

申请号：US18192211

申请日：2023-03-29

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Xiaozhe REN ,  Yichun YIN ,  Xin JIANG

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: This application relates to the field of artificial intelligence, and provides a model training method. The method includes: obtaining a to-be-trained first neural network model, where the first neural network model includes a first operator, and the first operator is used to perform a product operation on input data and a target weight matrix; replacing the first operator in the first neural network model with a second operator, to obtain a second neural network model, where the second operator is used to perform a product operation on input data and a plurality of sub-weight matrices, and the plurality of sub-weight matrices are obtained by performing matrix factorization on the target weight matrix; and performing model training on the second neural network model to obtain a target neural network model.