公开(公告)号：US20160011901A1

公开(公告)日：2016-01-14

申请号：US14327338

申请日：2014-07-09

Applicant: Google Inc.

Inventor： Jeremy S. Hurwitz ,  Eugene R. Kirpichov ,  Jelena Pjesivac-Grbovic ,  Kenneth J. Goldman

IPC: G06F9/46

CPC classification number: G06F9/46 ,  G06F9/4843

Abstract: The present disclosure relates to dynamically adjusting shard allocation during parallel processing operations. One example method includes determining a target completion time for a batch data processing job of an input data set performed by a plurality of tasks, each of the plurality of tasks processing a different input shard including a different portion of the input data set; identifying a first task having an estimated completion time greater than the target completion time of the batch data processing job; and splitting the first input shard into a first split input shard and a second split input shard different from the first split input shard, the first split input shard including a first portion of the first input shard, and the second split input shard including a second portion of the first input shard different from the first portion.

Abstract translation: 本公开涉及在并行处理操作期间动态地调整分片分配。 一个示例性方法包括确定由多个任务执行的输入数据集的批量数据处理作业的目标完成时间，所述多个任务中的每一个处理包括所述输入数据集的不同部分的不同输入分片; 识别具有大于批量数据处理作业的目标完成时间的估计完成时间的第一任务; 以及将所述第一输入碎片分割成与所述第一分割输入分片不同的第一分割输入分片和第二分割输入分片，所述第一分割输入分片包括所述第一输入分片的第一部分，所述第二分割输入分片包括第二分割输入分片 所述第一输入片与所述第一部分不同的部分。