公开(公告)号：US20180322391A1

公开(公告)日：2018-11-08

申请号：US15971884

申请日：2018-05-04

Applicant: NVIDIA Corporation

Inventor： Hao WU ,  Jonah ALBEN ,  Paulius MICIKEVICIUS

IPC: G06N3/08 ,  G06N3/04

CPC classification number: G06N3/084 ,  G06N3/04

Abstract: In training a deep neural network using reduced precision, gradient computation operates on larger values without affecting the rest of the training procedure. One technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision, and reduces the magnitude of the computed gradients to compensate for scaling of the loss. In one example non-limiting arrangement, the training forward pass scales a loss value by some factor S and the weight update reduces the weight gradient contribution by 1/S. Several techniques can be used for selecting scaling factor S and adjusting the weight update.