公开(公告)号：US20250165477A1

公开(公告)日：2025-05-22

申请号：US18511902

申请日：2023-11-16

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Alexander Balikov ,  Boyang Peng

IPC: G06F16/2455 ,  G06F9/48 ,  G06F16/2453

Abstract: A database system performs pipelined execution of queries that process batches of streaming data. The database system compiles a database query to generate an execution plan and determines a set of stages based on the execution plan. The database query processes streaming data comprising batches. A scheduler schedules pipelined execution stages of the database query. Accordingly, the database system performs execution of a particular stage processing a batch of the streaming data in parallel with subsequent stages of the database query processing previous batches of the streaming data. The system further maintains watermarks for different stages of the database query.