公开(公告)号：US20230409868A1

公开(公告)日：2023-12-21

申请号：US17844204

申请日：2022-06-20

Applicant: Advanced Micro Devices, Inc.

Inventor： Hai Xiao ,  Adam H Li ,  Harris Eleftherios Gasparakis

IPC: G06N3/04 ,  G06N3/08

CPC classification number: G06N3/04 ,  G06N3/08

Abstract: Activation scaled clipping layers for neural networks are described. An activation scaled clipping layer processes an output of a neuron in a neural network using a scaling parameter and a clipping parameter. The scaling parameter defines how numerical values are amplified relative to zero. The clipping parameter specifies a numerical threshold that causes the neuron output to be expressed as a value defined by the numerical threshold if the neuron output satisfies the numerical threshold. In some implementations, the scaling parameter is linear and treats numbers within a numerical range as being equivalent, such that any number in the range is scaled by a defined magnitude, regardless of value. Alternatively, the scaling parameter is nonlinear, which causes the activation scaled clipping layer to amplify numbers within a range by different magnitudes. Each scaling and clipping parameter is learnable during training of a machine learning model implementing the neural network.