公开(公告)号：US20240256539A1

公开(公告)日：2024-08-01

申请号：US18160850

申请日：2023-01-27

Applicant: Databricks, Inc.

Inventor： Shoumik Palkar ,  Alexander Behm ,  Mostafa Mokhtar ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F16/22

CPC classification number: G06F16/24539 ,  G06F16/221

Abstract: Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. The method includes receiving a request to perform a new query in a columnar database containing a plurality of columns. A step in the method includes accessing a set of data in a column of the plurality of columns based on the query. The method includes generating an input to a machine-learned model comprising characteristics of the set of data in the column. From the machine-learned model, the method includes generating a likelihood value indicative of whether a filter of a first portion of the set of data in the column has greater efficiency than a download followed by a filter of the set of data in the column. The method further includes comparing the likelihood value to a threshold value. Based on the comparison, the method includes filtering the first portion of the set of data before downloading the set of data if the likelihood value is equal to or above the threshold value.

公开(公告)号：US20240256543A1

公开(公告)日：2024-08-01

申请号：US18160861

申请日：2023-01-27

Applicant: Databricks, Inc.

Inventor： Shoumik Palkar ,  Alexander Behm ,  Mostafa Mokhtar ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24545 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. A data processing service receives a request to perform a query identifying a filter column and a non-filter column in a columnar database. The data processing service accesses a first task of contiguous rows in the filter column from a cloud-based object storage. The data processing service applies a filter defined by the query to the first task. The data processing service generates filter results for the first task that may include a percentage of the first task discarded and a run-time. The data processing service determines, based on the filter results for the first task, a likelihood value that indicates a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query.

公开(公告)号：US20230161767A1

公开(公告)日：2023-05-25

申请号：US18158258

申请日：2023-01-23

Applicant: Databricks, Inc.

Inventor： Shi Xin ,  Alexander Behm ,  Shoumik Palkar ,  Herman Rudolf Petrus Catharina van Hovell tot Westerflier

IPC: G06F16/2453 ,  G06F16/2458 ,  G06F16/25

CPC classification number: G06F16/24542 ,  G06F16/258 ,  G06F16/2471

Abstract: A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.

公开(公告)号：US20240256550A1

公开(公告)日：2024-08-01

申请号：US18162616

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.

公开(公告)号：US20240256549A1

公开(公告)日：2024-08-01

申请号：US18162607

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24558 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.

公开(公告)号：US12242485B2

公开(公告)日：2025-03-04

申请号：US18162616

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/24 ,  G06F11/34 ,  G06F16/22 ,  G06F16/2455

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least a operator and a request to return information about a value of interest in a columnar dataset stored on cloud storage. At least one column in the columnar dataset is based on a dictionary. The dictionary maps one or more values for a column to one or more respective identifiers. The method determines whether to perform dictionary filtering for the query by calculating a metric based on one or more factors. Responsive to the metric being below a threshold, which may be predetermined, the method performs the dictionary filtering.

公开(公告)号：US12210528B2

公开(公告)日：2025-01-28

申请号：US18162607

申请日：2023-01-31

Applicant: Databricks, Inc.

Inventor： Utkarsh Agarwal ,  Shoumik Palkar ,  Alexander Behm ,  Sriram Krishnamurthy

IPC: G06F16/2455 ,  G06F11/34 ,  G06F16/22

Abstract: Disclosed herein is a method, system, or non-transitory computer readable medium for evaluating a query on a columnar dataset comprising one or more dictionaries associated with columns in the dataset. The method includes receiving a request to perform a query comprising at least an operator for a columnar dataset on cloud storage. At least one column in the dataset is based on a dictionary, and the dictionary maps one or more values for a column to one or more respective identifiers. The method evaluates the operator on one or more values of the dictionary to generate an updated dictionary comprising updated values. The method may decode the updated dictionary into an updated column comprising updated data values.

公开(公告)号：US12124450B2

公开(公告)日：2024-10-22

申请号：US18160861

申请日：2023-01-27

Applicant: Databricks, Inc.

Inventor： Shoumik Palkar ,  Alexander Behm ,  Mostafa Mokhtar ,  Sriram Krishnamurthy

IPC: G06F16/2453 ,  G06F11/34 ,  G06F16/22

CPC classification number: G06F16/24545 ,  G06F11/3409 ,  G06F16/221

Abstract: Disclosed herein is a method for determining whether to apply a lazy materialization technique to a query run. A data processing service receives a request to perform a query identifying a filter column and a non-filter column in a columnar database. The data processing service accesses a first task of contiguous rows in the filter column from a cloud-based object storage. The data processing service applies a filter defined by the query to the first task. The data processing service generates filter results for the first task that may include a percentage of the first task discarded and a run-time. The data processing service determines, based on the filter results for the first task, a likelihood value that indicates a likelihood of gaining a performance benefit by applying the lazy materialization technique to a second task of the query.

公开(公告)号：US11874832B2

公开(公告)日：2024-01-16

申请号：US18158258

申请日：2023-01-23

Applicant: Databricks, Inc.

Inventor： Shi Xin ,  Alexander Behm ,  Shoumik Palkar ,  Herman Rudolf Petrus Catharina van Hovell tot Westerflier

IPC: G06F16/2453 ,  G06F16/2458 ,  G06F16/25

CPC classification number: G06F16/24542 ,  G06F16/2471 ,  G06F16/258

Abstract: A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.

公开(公告)号：US11586624B2

公开(公告)日：2023-02-21

申请号：US17237979

申请日：2021-04-22

Applicant: Databricks Inc.

Inventor： Shi Xin ,  Alexander Behm ,  Shoumik Palkar ,  Herman Rudolf Petrus Catharina van Hövell tot Westerflier

IPC: G06F16/2453 ,  G06F16/2458 ,  G06F16/25

Abstract: A system comprises an interface, a processor, and a memory. The interface is configured to receive a query. The processor is configured to: determine a set of nodes for the query; determine whether a node of the set of nodes comprises a first engine node type or a second engine node type, wherein determining whether the node of the set of nodes comprises the first engine node type or the second engine node type is based at least in part on determining whether the node is able to be executed in a second engine; and generate a plan based at least in part on the set of nodes. The memory is coupled to the processor and is configured to provide the processor with instructions.