Heartbeat propagation in a distributed stream processing system

    公开(公告)号:US11507585B2

    公开(公告)日:2022-11-22

    申请号:US17124318

    申请日:2020-12-16

    IPC分类号: G06F16/2455 G06F16/2458

    摘要: An event processing system for distributing a continuous query, and processing events in an event stream, is disclosed. The system can synchronize timestamps across all partitions by adding a heartbeat propagator component propagating a heartbeat signal for silent partitions. The system can further include a timestamp tracker component that tracks the maximum observed timestamp across all partitions for each batch, and sends this to each partition. The system can distribute partitions of the events to the worker nodes, and instruct a heartbeat propagator component to generate a heartbeat for a first worker node corresponding to an event in a different partition. The system receives outputs from the worker nodes and combines these into a single result corresponding to a continuous query. The system can further implement a timestamp tracker to select a highest timestamp value of a previous batch, and send the value to the worker nodes.

    Integrating logic in micro batch based event processing systems

    公开(公告)号:US11503107B2

    公开(公告)日:2022-11-15

    申请号:US16950127

    申请日:2020-11-17

    摘要: A distributed event processing system is disclosed that receives continuous data streams, registers a continuous query against the data streams, and continuously executes the query as new data appears in the streams. In certain embodiments, the distributed event processing system deploys and executes applications (e.g., event processing applications) by distributing the execution of the application on a cluster of machines within the system. In certain embodiments, the system provides users with the ability to specify logical rules in an application. The system processes the logical rules by generating a DAG of transformations representing the logical rules and converts the DAG of transformations into a logical rules Resilient Distributed Dataset (RDD) DAG of transformations. The system processes events in an event batch against the RDD DAG of transformations to generate a set of output results for the user.

    Processing timestamps and heartbeat events for automatic time progression

    公开(公告)号:US11061722B2

    公开(公告)日:2021-07-13

    申请号:US16653758

    申请日:2019-10-15

    IPC分类号: G06F9/48 G06F16/2455

    摘要: An event processing system for processing events in an event stream is disclosed. The system is configured for configuring a stream processor to micro-batch incoming events from a stream source. The system is also configured for generating a single timestamp for a micro-batch of the incoming events and/or receiving the micro-batch of the incoming events from the stream source. The system can also be configured for assigning the single timestamp to each event of the micro-batch and/or generating separate timestamp values for each respective event of the micro-batch. In some examples, the system can also be configured for assigning, for each respective event of the micro-batch, an individual one of the separate timestamp values.

    Heartbeat propagation in a distributed stream processing system

    公开(公告)号:US10902000B2

    公开(公告)日:2021-01-26

    申请号:US16121490

    申请日:2018-09-04

    IPC分类号: G06F16/2455 G06F16/2458

    摘要: An event processing system for distributing a continuous query, and processing events in an event stream, is disclosed. The system can synchronize timestamps across all partitions by adding a heartbeat propagator component propagating a heartbeat signal for silent partitions. The system can further include a timestamp tracker component that tracks the maximum observed timestamp across all partitions for each batch, and sends this to each partition. The system can distribute partitions of the events to the worker nodes, and instruct a heartbeat propagator component to generate a heartbeat for a first worker node corresponding to an event in a different partition. The system receives outputs from the worker nodes and combines these into a single result corresponding to a continuous query. The system can further implement a timestamp tracker to select a highest timestamp value of a previous batch, and send the value to the worker nodes.

    Fault-tolerant stream processing
    5.
    发明授权

    公开(公告)号:US10831619B2

    公开(公告)日:2020-11-10

    申请号:US16145575

    申请日:2018-09-28

    IPC分类号: G06F11/20 G06F11/16

    摘要: Techniques for providing fault-tolerant stream processing. An exemplary technique includes writing primary output events to a primary target and secondary output events to one or more secondary targets, where the primary output events are written by a primary server and the secondary output events are written by one or more secondary servers. The technique further includes receiving an election of a new primary server from a synchronization system upon a failure of the primary server, where the new primary server is elected from the one or more secondary servers. The technique further includes determining, by the new primary server, the primary output events that failed to be written to the primary target because of the failure of the primary server, and writing, by the new primary server, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

    Event batching, output sequencing, and log based state storage in continuous query processing

    公开(公告)号:US10740196B2

    公开(公告)日:2020-08-11

    申请号:US16279855

    申请日:2019-02-19

    摘要: An event processing system is disclosed that processes events of an event stream and performs the recovery of events during system failure. The recovery of events is performed by generating output sequence numbers for each deterministic output event generated by the event processing system. In an embodiment, the event processing system determines a current output sequence number of a most recently transmitted output event in a output stream of events, a last output sequence number of an output event corresponding to a most recently processed batch of input events and determines a set of one or more output events of the output stream to be transmitted based on the current sequence number and the last output sequence number. The event processing system then transmits the output events related to the application.