公开(公告)号：US20250117145A1

公开(公告)日：2025-04-10

申请号：US18990923

申请日：2024-12-20

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Xiaoming BAO ,  Hua XU ,  Chun LIU ,  Minjun ZHOU ,  Jiahuai CHEN

IPC: G06F3/06

Abstract: This disclosure provides data processing methods, apparatuses, and systems. In an implementation, a method comprises receiving, by a first data processing unit (DPU), a memory allocation request from a first service process in a computing node that comprises the first DPU, wherein the first DPU stores memory address assignment information that indicates a virtual memory address corresponding to a physical memory allocated by a storage node to the computing node. Determining, by the first DPU based on the memory address assignment information and in the virtual memory address corresponding to the physical memory allocated by the storage node to the computing node, a virtual memory address range assigned to the first service process, and when the first DPU receives a data storage request from the first service process, sending, by the first DPU, a write data request to the second DPU over a communication link between the first DPU and the second DPU.

公开(公告)号：US20250130726A1

公开(公告)日：2025-04-24

申请号：US19000111

申请日：2024-12-23

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Hua XU ,  Xiaoming BAO ,  Ce ZHU ,  Hongwei SUN ,  Xinglong WANG

IPC: G06F3/06

Abstract: Example data processing methods and apparatus are described. In one example method, a control node estimates a data volume of result data obtained after a plurality of second computing nodes execute a data processing task in parallel. The control node determines a quantity of reduce tasks based on the data volume and memory information of a first computing node that executes a reduce task. Each of the plurality of second computing nodes partitions, based on the quantity of reduced tasks, the result data generated after the data processing task is executed, where each partition corresponds to one reduce task. The first computing node performs reduce processing on data obtained after partitioning is performed by the plurality of second computing nodes.

公开(公告)号：US20240220334A1

公开(公告)日：2024-07-04

申请号：US18604395

申请日：2024-03-13

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Hua XU ,  Xiaoming BAO ,  Chun LIU ,  Minjun ZHOU ,  Jiahuai CHEN

IPC: G06F9/50

CPC classification number: G06F9/5083 ,  G06F9/5016 ,  G06F9/5038

Abstract: A data processing method in a distributed system is provided. The method includes: A management node splits a to-be-processed data processing request into a map task and a reduce task. A first worker node executes the map task, and writes intermediate data obtained by executing the map task into a shared storage area. The intermediate data is divided into a plurality of partitions based on a quantity of second worker nodes. Each partition belongs to one second worker node. An aggregation node stores, in a centralized manner, intermediate data that belongs to a same partition in the shared storage area. The second worker node obtains intermediate data of a corresponding partition to execute the reduce task.

公开(公告)号：US20240036728A1

公开(公告)日：2024-02-01

申请号：US18485847

申请日：2023-10-12

Applicant: HUAWEI TECHNOLOGIES CO., LTD.

Inventor： Hua XU ,  Xiaoming BAO ,  Hongwei SUN ,  Yihai ZHENG ,  Chun LIU

IPC: G06F3/06

CPC classification number: G06F3/0608 ,  G06F3/067 ,  G06F3/0644 ,  G06F3/064

Abstract: Methods and apparatuses for processing data are provided. In an implementation, a method comprising: obtaining, by a reduction server of a plurality of reduction servers in a distributed processing system and from a first storage area, metadata of first data to be read, wherein the first data comprises a target data block in a plurality of data blocks of second data, and wherein the second data comprises a processing result of a mapping server of a plurality of mapping servers for input data, determining a first address of the first data in a global memory based on the metadata, wherein the global memory comprises memories of the plurality of mapping servers and the plurality of reduction servers, and reading the first data from the global memory based on the first address.