公开(公告)号：US20240281667A1

公开(公告)日：2024-08-22

申请号：US18571151

申请日：2021-10-18

Applicant: Intel Corporation

Inventor： Guokai MA ,  Jiong GONG ,  Hongzhen LIU

IPC: G06N3/098

CPC classification number: G06N3/098

Abstract: Provided herein are apparatus and methods for batch rebalance in distributed data parallel DNN training. An apparatus includes interface circuitry; and processor circuitry coupled with the interface circuitry, wherein the processor circuitry is to: obtain sorted samples of a mini batch via the interface circuitry, wherein the sorted samples are in an ascend or descend order based on a volume of each of the samples; and assign the sorted samples to each of a plurality of local batches one by one in an order from a first local batch to a last local batch of the plurality of local batches and then from the last local batch to the first local batch until all of the sorted samples are assigned. Other embodiments may also be disclosed and claimed.