Abstract:
In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.
Abstract:
Embodiments of this application provide a network flow control method and a network device. The method includes: receiving a packet flow; determining, based on a service type of the packet flow, a service pipeline used for transmitting the packet flow, where service types of all packet flows in the service pipeline are the same; and based on a bandwidth weight allocated to the service type, transferring the packet flow in the service pipeline to a physical port. In the embodiments of this application, packet flows are allocated to different service pipelines based on a service type, and bandwidth weights are allocated, in a centralized manner, to service pipelines that carry a same service type.
Abstract:
In a data processing method, a worker node in a distributed data processing system receives first data from an upstream worker node. The first data has been stored in a buffer of the upstream worker node. The worker node sends a first portion of the first data to a persistent storage device of the distributed data processing system for persistent backup, and performs computational processing on the first data to generate second data. Prior to completing performing computational processing on the first data, the worker node sends acknowledgement information to the upstream worker node to instruct the upstream node to delete the first data from the buffer of the upstream worker node. The worker node then sends the second data to a downstream worker node in the distributed data processing system for further processing by the downstream worker node.
Abstract:
Embodiments of this application provide a network flow control method and a network device. The method includes: receiving a packet flow; determining, based on a service type of the packet flow, a service pipeline used for transmitting the packet flow, where service types of all packet flows in the service pipeline are the same; and based on a bandwidth weight allocated to the service type, transferring the packet flow in the service pipeline to a physical port. In the embodiments of this application, packet flows are allocated to different service pipelines based on a service type, and bandwidth weights are allocated, in a centralized manner, to service pipelines that carry a same service type.
Abstract:
One example packet processing device includes a buffer, and the packet processing device obtains a to-be-measured packet. In response to determining that occupied storage space in the buffer is less than a preset threshold, the packet processing device reads the to-be-measured information from the buffer, and modifies, based on the to-be-measured information and a first algorithm, a pieces of data in first measurement data corresponding to the to-be-measured packet, where a is a positive integer. In response to determining that occupied storage space in the buffer is greater than or equal to a preset threshold, the packet processing device modifies, based on to-be-measured information and a second algorithm, w pieces of data in second measurement data corresponding to the to-be-measured packet, where w is a positive integer, and w is less than a.
Abstract:
A method, an apparatus, and a system for identifying an abnormal IP data stream, which are used to improve identification accuracy. The method provided by the embodiments of the present invention includes: receiving Y elements sent by a data collection node; mapping the Y elements to N buckets; acquiring a bucket in the N buckets as a target bucket; acquiring r upper traffic limits of a first object in r buckets within the current time interval, the first object is any object mapped to the target bucket; and identifying, according to a preset abnormal object type and the r upper traffic limits within the current time interval, whether the first object is an abnormal object, where the preset abnormal object type is a heavy hitter or a heavy changer.
Abstract:
A network flow measurement method is applicable to a system including a network measurement device and a control plane device. The network flow measurement method includes measuring, by the network measurement device, first data, where the first data includes a first-type data structure, the first-type data structure includes first measurement information of a flow, and the first measurement information corresponds to a bit of a keyword of the flow, and sending, by the network measurement device, the first data to the control plane device.
Abstract:
A method, an apparatus, and a system for identifying an abnormal IP data stream, which are used to improve identification accuracy. The method provided by the embodiments of the present invention includes: receiving Y elements sent by a data collection node; mapping the Y elements to N buckets; acquiring a bucket in the N buckets as a target bucket; acquiring r upper traffic limits of a first object in r buckets within the current time interval, the first object is any object mapped to the target bucket; and identifying, according to a preset abnormal object type and the r upper traffic limits within the current time interval, whether the first object is an abnormal object, where the preset abnormal object type is a heavy hitter or a heavy changer.
Abstract:
A network flow measurement method is applicable to a system including a network measurement device and a control plane device. The network flow measurement method includes measuring, by the network measurement device, first data, where the first data includes a first-type data structure, the first-type data structure includes first measurement information of a flow, and the first measurement information corresponds to a bit of a keyword of the flow, and sending, by the network measurement device, the first data to the control plane device.