公开(公告)号：US11068447B2

公开(公告)日：2021-07-20

申请号：US15487896

申请日：2017-04-14

Applicant: Databricks Inc.

Inventor： Eric Keng-hao Liang ,  Srinath Shankar ,  Shi Xin

IPC: G06F16/18 ,  G06F16/16 ,  G06F16/182

Abstract: A system for directory level atomic commits includes an interface and a processor. The interface is configured to receive an indication to provide a set of files. The processor is configured to determine whether a file in a directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted and provide the file as one file of the set of files in the event that the file in the directory has been either 1) atomically committed or 2) written by a non-atomic process and not designated as deleted.

公开(公告)号：US10810051B1

公开(公告)日：2020-10-20

申请号：US16188989

申请日：2018-11-13

Applicant: Databricks Inc.

Inventor： Srinath Shankar ,  Eric Keng-Hao Liang

IPC: G06F9/46 ,  G06F9/50 ,  G06F9/48 ,  G06F9/38 ,  H04L29/08

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a set of tasks executing or pending on the set of cluster machines; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of the set of tasks nor storing one or more intermediate data files of a set of intermediate data files, where the set of intermediate data files is associated with a set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

公开(公告)号：US20200301684A1

公开(公告)日：2020-09-24

申请号：US16864074

申请日：2020-04-30

Applicant: Databricks Inc.

Inventor： Srinath Shankar ,  Eric Keng-Hao Liang ,  Gregory George Owen

IPC: G06F8/41 ,  G06F8/54 ,  G06F8/70 ,  G06F11/36 ,  G06F11/07 ,  G06F21/62 ,  G06F16/23 ,  G06F16/907

Abstract: A system for code development and execution includes a client interface and a client processor. The client interface is configured to receive user code for execution and receive an indication of a server that will perform the execution. The client processor is configured to parse the user code to identify one or more data items referred to during the execution. The client processor is also configured to provide the server with an inquiry for metadata regarding the one or more data items, receive the metadata regarding the one or more data items, determine a logical plan based at least in part on the metadata regarding the one or more data items; and provide the logical plan to the server for execution.

公开(公告)号：US20200241950A1

公开(公告)日：2020-07-30

申请号：US16793921

申请日：2020-02-18

Applicant: Databricks Inc.

Inventor： Alicja Luszczak ,  Srinath Shankar ,  Shi Xin

IPC: G06F11/07 ,  G06F11/34 ,  G06F11/30

Abstract: A system for monitoring job execution includes an interface and a processor. The interface is configured to receive an indication to start a cluster processing job. The processor is configured to determine whether processing a data instance associated with the cluster processing job satisfies a watchdog criterion; and in the event that processing the data instance satisfies the watchdog criterion, cause the processing of the data instance to be killed.

公开(公告)号：US11379272B2

公开(公告)日：2022-07-05

申请号：US17020573

申请日：2020-09-14

Applicant: Databricks Inc.

Inventor： Srinath Shankar ,  Eric Keng-Hao Liang

IPC: G06F9/46 ,  G06F9/50 ,  G06F9/48 ,  G06F9/38 ,  H04L67/5682

Abstract: The allocation system comprises an interface and a processor. The interface is configured to receive an indication to deactivate idle cluster machines of a set of cluster machines. The processor is configured to determine a list of cluster machines storing one or more intermediate data files of a set of intermediate data files; determine a set of idle cluster machines of the set of cluster machines that are neither running one or more tasks of a set of tasks executing or pending on the set of cluster machines nor storing the one or more intermediate data files of the set of intermediate data files, where the set of intermediate data files is associated with the set of tasks executing or pending on the cluster machines; and deactivate each cluster machine of the set of idle cluster machines.

公开(公告)号：US10474501B2

公开(公告)日：2019-11-12

申请号：US15581987

申请日：2017-04-28

Applicant: Databricks Inc.

Inventor： Ali Ghodsi ,  Srinath Shankar ,  Sameer Paranjpye ,  Shi Xin ,  Matei Zaharia

IPC: G06F9/50

Abstract: A system for cluster resource allocation includes an interface and a processor. The interface is configured to receive a process and input data. The processor is configured to determine an estimate for resources required for the process to process the input data; determine existing available resources in a cluster for running the process; determine whether the existing available resources are sufficient for running the process; in the event it is determined that the existing available resources are not sufficient for running the process, indicate to add new resources; determine an allocated share of resources in the cluster for running the process; and cause execution of the process using the share of resources.