公开(公告)号：US20170147594A1

公开(公告)日：2017-05-25

申请号：US15423684

申请日：2017-02-03

Applicant: Huawei Technologies Co., Ltd.

Inventor： Hongli Huang ,  Yanchu Liu

IPC: G06F17/30

CPC classification number: G06F16/162 ,  G06F16/00 ,  G06F16/254

Abstract: Heterogeneous data source shielding by an integrated development environment (IDE), including receiving a product process release request or a product process test request initiated by a user, obtaining a configuration parameter of a product and a data flow model preconfigured for the product, where the data flow model includes a multi-input node that connects multiple input data sources, and a matching relationship between the input data sources and different configuration parameters is configured on the multi-input node looking up a corresponding input data source according to the configuration parameter of the product, replacing the multi-input node in the data flow model with the corresponding input data source, and using a data flow model obtained after replacing for the product process release or the product process test.

公开(公告)号：US20160364273A1

公开(公告)日：2016-12-15

申请号：US15248085

申请日：2016-08-26

Applicant: Huawei Technologies Co., Ltd.

Inventor： Yanchu Liu ,  Chao Liu

IPC: G06F9/50

CPC classification number: G06F9/5066

Abstract: Embodiments of the present disclosure disclose a data processing apparatus and method for processing serial tasks, which can reduce, in a data processing process, a quantity of times of reading an output result from disks of node devices in a distributed system, thereby reducing time required and network resources occupied for processing a large amount of data. The method in the present disclosure includes: acquiring at least two MapReduce tasks, where the at least two MapReduce tasks are serially arranged according to an execution sequence, and when two MapReduce tasks are serially arranged, an output value obtained after the former MapReduce task is executed is an input value of the latter MapReduce task; combining the at least two MapReduce tasks to obtain a target MapReduce task; and executing the target MapReduce task, and obtaining an output result of the target MapReduce task.

Abstract translation: 本公开的实施例公开了一种用于处理串行任务的数据处理装置和方法，其可以在数据处理过程中减少从分布式系统中的节点设备的盘读取输出结果的次数，从而减少所需的时间 并占用了处理大量数据的网络资源。 本公开的方法包括：获取至少两个MapReduce任务，其中至少两个MapReduce任务根据执行顺序被串行排列，并且当两个MapReduce任务被串行排列时，在前MapReduce任务之后获得的输出值是 被执行的是后面MapReduce任务的输入值; 组合至少两个MapReduce任务以获得目标MapReduce任务; 并执行目标MapReduce任务，并获得目标MapReduce任务的输出结果。