公开(公告)号：US20240256543A1

公开(公告)日：2024-08-01

申请号：US18160861

申请日：2023-01-27

Applicant: Databricks, Inc.

Inventor： Shoumik Palkar ,  Alexander Behm ,  Mostafa Mokhtar ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24545 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. A data processing service receives a request to perform a query identifying a filter column and a non-filter column in a columnar database. The data processing service accesses a first task of contiguous rows in the filter column from a cloud-based object storage. The data processing service applies a filter defined by the query to the first task. The data processing service generates filter results for the first task that may include a percentage of the first task discarded and a run-time. The data processing service determines, based on the filter results for the first task, a likelihood value that indicates a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query.

公开(公告)号：US20230161767A1

公开(公告)日：2023-05-25

申请号：US18158258

申请日：2023-01-23

Applicant: Databricks, Inc.

Inventor： Shi Xin ,  Alexander Behm ,  Shoumik Palkar ,  Herman Rudolf Petrus Catharina van Hovell tot Westerflier

IPC: G06F16/2453 ,  G06F16/2458 ,  G06F16/25

CPC classification number: G06F16/24542 ,  G06F16/258 ,  G06F16/2471

Abstract: A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.

公开(公告)号：US12153558B1

公开(公告)日：2024-11-26

申请号：US18162093

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Alexander Behm ,  Ankur Dave

IPC: G06F16/00 ,  G06F16/13 ,  G06F16/22 ,  G06F16/242 ,  G06F16/2455 ,  G06F16/28

Abstract: A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.

公开(公告)号：US20240256539A1

公开(公告)日：2024-08-01

申请号：US18160850

申请日：2023-01-27

Applicant: Databricks, Inc.

Inventor： Shoumik Palkar ,  Alexander Behm ,  Mostafa Mokhtar ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/22

CPC classification number: G06F16/24539 ,  G06F16/221

Abstract: Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.

公开(公告)号：US12072880B2

公开(公告)日：2024-08-27

申请号：US17892376

申请日：2022-08-22

Applicant: Databricks, Inc.

Inventor： Prashanth Menon ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F9/00 ,  G06F16/2453 ,  G06F16/28

CPC classification number: G06F16/24542 ,  G06F16/285

Abstract: The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.

公开(公告)号：US20240256550A1

公开(公告)日：2024-08-01

申请号：US18162616

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.

公开(公告)号：US20240256549A1

公开(公告)日：2024-08-01

申请号：US18162607

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.

公开(公告)号：US20240061840A1

公开(公告)日：2024-02-22

申请号：US18162366

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Prashanth Menon ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/28

CPC classification number: G06F16/24542 ,  G06F16/285

Abstract: The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.

公开(公告)号：US20240061839A1

公开(公告)日：2024-02-22

申请号：US17892376

申请日：2022-08-22

Applicant: Databricks, Inc.

Inventor： Prashanth Menon ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/28

CPC classification number: G06F16/24542 ,  G06F16/285

Abstract: The present application discloses a method, system, and computer system for parsing files. The method includes receiving an indication that a first file is to be processed, determining to begin processing the first file using a first processing engine based at least in part on one or more predefined heuristics, indicating to process the first file using a first processing engine, determining whether a particular error in processing the first file using the first processing engine has been detected, in response to determining that the particular error has been detected, indicate to stop processing the first file using the first processing engine and indicate to continue processing using a second processing engine, and storing in memory information obtained based on processing the first file by one or more of the first processing engine and the second processing engine.

公开(公告)号：US11675767B1

公开(公告)日：2023-06-13

申请号：US17099467

申请日：2020-11-16

Applicant: Databricks, Inc.

Inventor： Alexander Behm ,  Ankur Dave

IPC: G06F16/00 ,  G06F16/22 ,  G06F16/28 ,  G06F16/242 ,  G06F16/2455 ,  G06F16/13

CPC classification number: G06F16/2255 ,  G06F16/134 ,  G06F16/2272 ,  G06F16/244 ,  G06F16/24556 ,  G06F16/285

Abstract: A system includes a plurality of computing units. A first computing unit of the plurality of computing units comprises: a communication interface configured to receive an indication to roll up data in a data table; and a processor coupled to the communication interface and configured to: build a preaggregation hash table based at least in part on a set of columns and the data table by aggregating input rows of the data table; for each preaggregated hash table entry of the preaggregated hash table: provide the preaggregated hash table entry to a second computing unit of the plurality of computing units based at least in part on a distribution hash value; receive a set of received entries from computing units of the plurality of computing units; and build an aggregation hash table based at least in part on the set of received entries by aggregating the set of received entries.