公开(公告)号：US20250086177A1

公开(公告)日：2025-03-13

申请号：US18745847

申请日：2024-06-17

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F16/2453 ,  G06F16/2455

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.

公开(公告)号：US20230161767A1

公开(公告)日：2023-05-25

申请号：US18158258

申请日：2023-01-23

Applicant: Databricks, Inc.

Inventor： Shi Xin ,  Alexander Behm ,  Shoumik Palkar ,  Herman Rudolf Petrus Catharina van Hovell tot Westerflier

IPC: G06F16/2453 ,  G06F16/2458 ,  G06F16/25

CPC classification number: G06F16/24542 ,  G06F16/258 ,  G06F16/2471

Abstract: A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.

公开(公告)号：US20180314732A1

公开(公告)日：2018-11-01

申请号：US15581647

申请日：2017-04-28

Applicant: Databricks Inc.

Inventor： Michael Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F17/30

CPC classification number: G06F16/24542 ,  G06F16/24568

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.

公开(公告)号：US20180314556A1

公开(公告)日：2018-11-01

申请号：US15581987

申请日：2017-04-28

Applicant: Databricks Inc.

Inventor： Ali Ghodsi ,  Srinath Shankar ,  Sameer Paranjpye ,  Shi Xin ,  Matei Zaharia

IPC: G06F9/50

CPC classification number: G06F9/5061 ,  G06F2209/5011 ,  G06F2209/505

Abstract: A system for cluster resource allocation includes an interface and a processor. The interface is configured to receive a process and input data. The processor is configured to determine an estimate for resources required for the process to process the input data; determine existing available resources in a cluster for running the process; determine whether the existing available resources are sufficient for running the process; in the event it is determined that the existing available resources are not sufficient for running the process, indicate to add new resources; determine an allocated share of resources in the cluster for running the process; and cause execution of the process using the share of resources.

公开(公告)号：US20180300354A1

公开(公告)日：2018-10-18

申请号：US15487896

申请日：2017-04-14

Applicant: Databricks Inc.

Inventor： Eric Keng-hao Liang ,  Srinath Shankar ,  Shi Xin

IPC: G06F17/30

Abstract: A system for directory level atomic commits includes an interface and a processor. The interface is configured to receive an indication to provide a set of files. The processor is configured to determine whether a file in a directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted and provide the file as one file of the set of files in the event that the file in the directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted.

公开(公告)号：US20230359516A1

公开(公告)日：2023-11-09

申请号：US18200316

申请日：2023-05-22

Applicant: Databricks, Inc.

Inventor： Alicja Luszczak ,  Srinath Shankar ,  Shi Xin

IPC: G06F11/07 ,  G06F11/34 ,  G06F11/30

CPC classification number: G06F11/0757 ,  G06F11/0721 ,  G06F11/0793 ,  G06F11/3419 ,  G06F11/3024 ,  G06F11/076 ,  G06F2201/88 ,  G06F2201/81

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

公开(公告)号：US11599783B1

公开(公告)日：2023-03-07

申请号：US15610062

申请日：2017-05-31

Applicant: Databricks, Inc.

Inventor： Sue Ann Hong ,  Shi Xin ,  Timothee Hunter ,  Ali Ghodsi

IPC: G06N3/08 ,  G06N3/063 ,  G06N5/02 ,  G06N3/04 ,  G06N5/022 ,  G06F16/14 ,  G06F16/22

Abstract: A function creation method is disclosed. The method comprises defining one or more database function inputs, defining cluster processing information, defining a deep learning model, and defining one or more database function outputs. A database function is created based at least in part on the one or more database function inputs, the cluster set-up information, the deep learning model, and the one or more database function outputs. In some embodiments, the database function enables a non-technical user to utilize deep learning models.

公开(公告)号：US20220083410A1

公开(公告)日：2022-03-17

申请号：US17537124

申请日：2021-11-29

Applicant: Databricks Inc.

Inventor： Alicja Luszczak ,  Srinath Shankar ,  Shi Xin

IPC: G06F11/07 ,  G06F11/34 ,  G06F11/30

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

公开(公告)号：US11693723B2

公开(公告)日：2023-07-04

申请号：US17537124

申请日：2021-11-29

Applicant: Databricks, Inc.

Inventor： Alicja Luszczak ,  Srinath Shankar ,  Shi Xin

IPC: G06F11/07 ,  G06F11/34 ,  G06F11/30

CPC classification number: G06F11/0757 ,  G06F11/076 ,  G06F11/0721 ,  G06F11/0793 ,  G06F11/3024 ,  G06F11/3419 ,  G06F2201/81 ,  G06F2201/88

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

公开(公告)号：US20230141556A1

公开(公告)日：2023-05-11

申请号：US17976361

申请日：2022-10-28

Applicant: Databricks, Inc.

Inventor： Michael Paul Armbrust ,  Tathagata Das ,  Shi Xin ,  Matei Zaharia

IPC: G06F16/2453 ,  G06F16/2455

CPC classification number: G06F16/24542 ,  G06F16/24568

Abstract: A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.