公开(公告)号：US20190332944A1

公开(公告)日：2019-10-31

申请号：US16425012

申请日：2019-05-29

Applicant: Huawei Technologies Co., Ltd.

Inventor： Xiaolong Bai ,  Changzheng Zhang ,  Mingzhen Xia

IPC: G06N3/08 ,  G06N3/04

Abstract: A training method, apparatus, and chip for a neural network model includes determining a model training mode of each layer based on an estimated data volume in a model parameter set and an estimated data volume of output data of the layer, obtaining second output data that is obtained by m worker modules by training a (j−1)th layer, and directly obtaining by a worker module a global gradient of a model parameter by training the model parameter based on the second output data when a model parallel training mode is used for a jth layer.

公开(公告)号：US20210342696A1

公开(公告)日：2021-11-04

申请号：US17376722

申请日：2021-07-15

Applicant: Huawei Technologies Co., Ltd.

Inventor： Xiaolong Bai ,  Pengfei Li ,  Zhenyu Zhang

IPC: G06N3/08 ,  G06N3/04

Abstract: A deep learning model training method includes generating N first gradient sets in a back propagation (BP) calculation in a jth iteration of N deep learning models, adjusting a communication sequence of gradients included in each of the first gradient sets to obtain an adjusted communication sequence, and sending, according to the adjusted communication sequence, the gradients included in each of the N first gradient sets to the parameter storage space.