Trace messaging for distributed execution of data processing pipelines
Abstract:
A method for routing trace messages during an execution of a data processing pipeline is provided. The method may include receiving, at a master node, a request from a client to receive trace messages output by a worker node executing at least a portion of the data processing pipeline. The master node and the worker node may be part of a distributed cluster of computing nodes. The trace messages may correspond to events occurring during the execution of at least the portion of the data processing pipeline. The master node may respond to the request by subscribing to a trace stream published by the worker node. The trace stream may include the trace messages output by the worker node. The master node may generate a user interface for displaying, at the client, the trace messages. Related systems and articles of manufacture, including computer program products, are also provided.
Information query
Patent Agency Ranking
0/0