公开(公告)号：US20250086177A1

公开(公告)日：2025-03-13

申请号：US18745847

申请日：2024-06-17

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F16/2453 ,  G06F16/2455

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.

公开(公告)号：US20250013644A1

公开(公告)日：2025-01-09

申请号：US18769269

申请日：2024-07-10

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Tom Van Bussel ,  Prakhar Jain

IPC: G06F16/2453 ,  G06F11/34 ,  G06F16/22 ,  G06F16/28

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, obtaining one or more other resulting files based at least in part on unmatched rows, and obtaining a set of processed files based at least in part on performing a post-processing operation with respect to the set of resulting files. The set of processed files has less files than the set of resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s).

公开(公告)号：US20180314732A1

公开(公告)日：2018-11-01

申请号：US15581647

申请日：2017-04-28

Applicant: Databricks Inc.

Inventor： Michael Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F17/30

CPC classification number: G06F16/24542 ,  G06F16/24568

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.

公开(公告)号：US12045220B2

公开(公告)日：2024-07-23

申请号：US17895890

申请日：2022-08-25

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Chirstos Stavrakakis

IPC: G06F17/30 ,  G06F9/48 ,  G06F16/22

CPC classification number: G06F16/2282 ,  G06F9/4881

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and persist, in one or more deletion vector files, one or more deletion vectors for corresponding rows of the one or more target table files, and obtaining a resulting table based at least in part on the second job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and one or more deletion of vectors associated with previously removed rows of the matching target table files and obtaining the second job resulting file(s).

公开(公告)号：US20240070138A1

公开(公告)日：2024-02-29

申请号：US17895890

申请日：2022-08-25

Applicant: Databricks Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Chirstos Stavrakakis

IPC: G06F16/22 ,  G06F9/48

CPC classification number: G06F16/2282 ,  G06F9/4881

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and persist, in one or more deletion vector files, one or more deletion vectors for corresponding rows of the one or more target table files, and obtaining a resulting table based at least in part on the second job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and one or more deletion of vectors associated with previously removed rows of the matching target table files and obtaining the second job resulting file(s).

公开(公告)号：US11567900B1

公开(公告)日：2023-01-31

申请号：US17384486

申请日：2021-07-23

Applicant: Databricks Inc.

Inventor： Rahul Shivu Mahadev ,  Burak Yavuz ,  Tathagata Das

IPC: G06F16/22 ,  G06F16/172

Abstract: The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.

公开(公告)号：US20240202211A1

公开(公告)日：2024-06-20

申请号：US18219314

申请日：2023-07-07

Applicant: Databricks, Inc.

Inventor： Alexander Balikov ,  Tathagata Das ,  Karthikeyan Ramasamy

IPC: G06F16/27 ,  G06F16/2455

CPC classification number: G06F16/278 ,  G06F16/24568

Abstract: A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.

公开(公告)号：US20240070155A1

公开(公告)日：2024-02-29

申请号：US17895882

申请日：2022-08-25

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Tom Van Bussel

IPC: G06F16/2455 ,  G06F16/22

CPC classification number: G06F16/2456 ,  G06F16/2282

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first and second jobs, and obtaining other resulting files based at least in part on a second set of unmatched rows among the target table and the source table that results from the first set of unmatched rows having been processed in the second job, and obtaining a resulting table based on (i) second job resulting file(s), and (ii) other resulting files. Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a first matching action based on matched rows and a second matching action based on a subset of unmatched rows.

公开(公告)号：US20240069863A1

公开(公告)日：2024-02-29

申请号：US17895872

申请日：2022-08-25

Applicant: Databricks, Inc.

Inventor： Bart Samwel ,  Tathagata Das ,  Lars Kroll ,  Yijia Cui ,  Juliusz Sompolski ,  Tom Van Bussel

IPC: G06F7/14 ,  G06F16/14 ,  G06F16/16

CPC classification number: G06F7/14 ,  G06F16/148 ,  G06F16/16

Abstract: A method, system, and computer system for performing an operation with respect to a target table are disclosed. The method includes performing first, second and a third jobs, and obtaining a resulting table based at least in part on the second job resulting file(s) and third job resulting file(s). Performing the first job includes determining a set of matching target table files and storing target table information indicating for each of the set of matching target table files, a particular set of rows having matching rows. Performing the second job includes performing a matching action based on matched rows and obtaining the second job resulting file(s). Performing the third job includes determining unmatched rows for target table files and storing the unmatched rows in third job resulting file(s).

公开(公告)号：US20250061132A1

公开(公告)日：2025-02-20

申请号：US18822023

申请日：2024-08-30

Applicant: Databricks, Inc.

Inventor： Alexander Balikov ,  Tathagata Das ,  Karthikeyan Ramasamy

IPC: G06F16/27 ,  G06F16/2455

Abstract: A data processing service performs a rebalancing process for rebalancing stateful tasks on a cluster computing system. In one instance, the method for rebalancing stateful tasks is performed such that the per-operator partitions are spread across available executors of a cluster of the cluster computing system with respect to one or more statistics of the tasks. In one instance, the method for rebalancing stateful tasks is also performed such that the total number of stateful tasks are balanced per executor as long as this rebalancing does not imbalance the per-operator placements. In this way, the processing of stateful tasks can be spread across multiple executors in a relatively uniform manner, even though there may be an upfront cost of breaking the local caching on an executor.