公开(公告)号：US20250086177A1

公开(公告)日：2025-03-13

申请号：US18745847

申请日：2024-06-17

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F16/2453 ,  G06F16/2455

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.

公开(公告)号：US10769130B1

公开(公告)日：2020-09-08

申请号：US15987215

申请日：2018-05-23

Applicant: Databricks Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/23 ,  G06F16/14 ,  G06F16/22

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.

公开(公告)号：US11775499B2

公开(公告)日：2023-10-03

申请号：US17695411

申请日：2022-03-15

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/14 ,  G06F16/22 ,  G06F16/23

CPC classification number: G06F16/2358 ,  G06F16/148 ,  G06F16/2282

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.

公开(公告)号：US20220309103A1

公开(公告)日：2022-09-29

申请号：US17362450

申请日：2021-06-29

Applicant: Databricks Inc.

Inventor： Michael Paul Armbrust ,  Andreas Neumann ,  Mukul Murthy ,  Jonathan Mio

IPC: G06F16/901 ,  G06F16/22 ,  G06F16/245

Abstract: A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.

公开(公告)号：US20230394029A1

公开(公告)日：2023-12-07

申请号：US18236516

申请日：2023-08-22

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/23 ,  G06F16/14 ,  G06F16/22

CPC classification number: G06F16/2358 ,  G06F16/148 ,  G06F16/2282

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.

公开(公告)号：US11567998B2

公开(公告)日：2023-01-31

申请号：US17362450

申请日：2021-06-29

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Andreas Neumann ,  Mukul Murthy ,  Jonathan Mio

IPC: G06F16/901 ,  G06F16/245 ,  G06F16/22

Abstract: A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.

公开(公告)号：US20220253424A1

公开(公告)日：2022-08-11

申请号：US17695411

申请日：2022-03-15

Applicant: Databricks Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/23 ,  G06F16/14 ,  G06F16/22

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.

公开(公告)号：US11308071B2

公开(公告)日：2022-04-19

申请号：US16941227

申请日：2020-07-28

Applicant: Databricks Inc.

Inventor： Michael Paul Armbrust ,  Shixiong Zhu ,  Burak Yavuz

IPC: G06F16/23 ,  G06F16/14 ,  G06F16/22

Abstract: A system includes an interface and a processor. The interface is configured to receive a table indication of a data table and to receive a transaction indication to perform a transaction. The processor is configured to determine a current position N in a transaction log; determine a current state of the metadata; determine a read set associated with a transaction; attempt to write an update to the transaction log associated with a next position N+1; in response to a transaction determination that a simultaneous transaction associated with the next position N+1 already exists, determine a set of updated files; and in response to a determination that there is not an overlap between the read set associated with the current transaction and the set of updated files associated with the simultaneous transaction, attempt to write the update to the transaction to the transaction log associated with a further position N+2.

公开(公告)号：US12019682B2

公开(公告)日：2024-06-25

申请号：US18089349

申请日：2022-12-27

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Andreas Neumann ,  Mukul Murthy ,  Jonathan Mio

IPC: G06F16/901 ,  G06F16/215 ,  G06F16/22 ,  G06F16/245

CPC classification number: G06F16/9024 ,  G06F16/215 ,  G06F16/2282 ,  G06F16/245

Abstract: A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries and/or commands. The processor is coupled to the communication interface and configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; determine the dataflow graph by determining in-line expressions for tables of the dataflow graph aggregating calculations associated with a subset of dataflow graph nodes designated as view nodes; and provide the dataflow graph.

公开(公告)号：US12008040B2

公开(公告)日：2024-06-11

申请号：US17362456

申请日：2021-06-29

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Andreas Neumann ,  Mukul Murthy ,  Jonathan Mio

IPC: G06F16/901 ,  G06F16/215 ,  G06F16/22 ,  G06F16/245

CPC classification number: G06F16/9024 ,  G06F16/215 ,  G06F16/2282 ,  G06F16/245

Abstract: A system for dataflow graph processing comprises a communication interface and a processor. The communication interface is configured receive an indication to generate a dataflow graph, wherein the indication includes a set of queries. The processor is coupled to the communication interface and is configured to: determine dependencies of each query in the set of queries on another query; determine a DAG of nodes based at least in part on the dependencies; insert a node in the DAG of nodes to generate an updated DAG to enforce an expectation; determine a dataflow graph based on the updated DAG; and provide the dataflow graph.