公开(公告)号：US12001428B2

公开(公告)日：2024-06-04

申请号：US18047872

申请日：2022-10-19

Applicant: Snowflake Inc.

Inventor： Xinzhu Cai ,  Florian Andreas Funke

IPC: G06F16/24 ,  G06F16/22 ,  G06F16/2453

CPC classification number: G06F16/24537 ,  G06F16/2255

Abstract: Provided herein are systems and methods for handling build-side skew. For example, a method includes computing a plurality of hash values for a join operation. The join operation uses a corresponding plurality of row sets. The plurality of hash values are sampled to detect a frequent hash value. A build-side row set is partitioned using the frequent hash value to generate a partitioned build-side row set. The build-side row set is selected from the plurality of row sets. The partitioned build-side row set is distributed to a plurality of hash-join-build (HJB) instances executing at a corresponding plurality of servers.

公开(公告)号：US11675815B1

公开(公告)日：2023-06-13

申请号：US18102681

申请日：2023-01-27

Applicant: Snowflake Inc.

Inventor： Thierry Cruanes ,  Benoit Dageville ,  Florian Andreas Funke ,  Peter Povinec

IPC: G06F16/2455 ,  H04L41/0896 ,  H04L67/1008 ,  G06F16/28 ,  H04L67/1097 ,  H04L41/5025 ,  G06F9/50 ,  H04L43/0817

CPC classification number: G06F16/283 ,  G06F9/5072 ,  G06F16/2455 ,  H04L41/0896 ,  H04L41/5025 ,  H04L67/1008 ,  H04L67/1097 ,  H04L43/0817

Abstract: A method for a multi-cluster warehouse includes allocating processing units as part of a data warehouse. The processing units access data within one or more databases in cloud storage resources. The method also includes providing one or more queries to each processing unit within the data warehouse. In response to the queries, each processing unit performs database operations on a particular portion of a database table. The method also includes monitoring a workload of the processing units to determine that a processing capacity of the processing units has reached a threshold processing capacity. The method also includes dynamically adding additional processing units to and removing processing units from the data warehouse as needed based on a workload of the processing units.

公开(公告)号：US11620313B2

公开(公告)日：2023-04-04

申请号：US17118201

申请日：2020-12-10

Applicant: SNOWFLAKE INC.

Inventor： Florian Andreas Funke ,  Peter Povinec ,  Thierry Cruanes ,  Benoit Dageville

IPC: G06F16/28 ,  H04L67/1097 ,  G06F16/2455 ,  H04L41/0896 ,  H04L67/1008 ,  H04L41/5025 ,  G06F9/50 ,  H04L43/0817

Abstract: A method for a multi-cluster warehouse includes allocating a plurality of compute clusters as part of a virtual warehouse. The compute clusters are used to access and perform queries against one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the plurality of compute clusters. Each of the plurality of compute clusters of the virtual warehouse receives a plurality of queries so that the computing load is spread across the different clusters. The method also includes dynamically adding compute clusters to and removing compute clusters from the virtual warehouse as needed based on a workload of the plurality of compute clusters.

公开(公告)号：US11620287B2

公开(公告)日：2023-04-04

申请号：US16939750

申请日：2020-07-27

Applicant: Snowflake Inc.

Inventor： Bowei Chen ,  Thierry Cruanes ,  Florian Andreas Funke ,  Allison Waingold Lee ,  Jiaqi Yan

IPC: G06F16/2453 ,  G06F16/242

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation. The subject technology analyzes the at least one aggregation to generate a modified query plan, the modified query plan including at least a top aggregation operator, an intermediate aggregation operator, and a bottom aggregation operator. The subject technology performs, with respect to the intermediate aggregation operator, at least one operation comprising: the subject technology receives an input intermediate data type; the subject technology performs an internalize operation on the input intermediate data type to generate an internal state; the subject technology performs an accumulate operation on the internal state to generate intermediate data; and the subject technology performs an externalize operation on the intermediate data to generate an output data type.

公开(公告)号：US11615118B2

公开(公告)日：2023-03-28

申请号：US16862140

申请日：2020-04-29

Applicant: Snowflake Inc.

Inventor： Thierry Cruanes ,  Benoit Dageville ,  Florian Andreas Funke ,  Peter Povinec

IPC: G06F16/28 ,  H04L67/1097 ,  G06F16/2455 ,  H04L41/0896 ,  H04L67/1008 ,  H04L41/5025 ,  G06F9/50 ,  H04L43/0817

Abstract: A method for a multi-cluster warehouse includes allocating processing units as part of a data warehouse. The processing units access data within one or more databases in cloud storage resources. The method also includes providing one or more queries to each processing unit within the data warehouse. In response to the queries, each processing unit performs database operations on a particular portion of a database table. The method also includes monitoring a workload of the processing units to determine that a processing capacity of the processing units has reached a threshold processing capacity. The method also includes dynamically adding additional processing units to and removing processing units from the data warehouse as needed based on a workload of the processing units.

公开(公告)号：US11593404B2

公开(公告)日：2023-02-28

申请号：US16863758

申请日：2020-04-30

Applicant: Snowflake Inc.

Inventor： Thierry Cruanes ,  Benoit Dageville ,  Florian Andreas Funke ,  Peter Povinec

IPC: G06F16/28 ,  H04L67/1097 ,  G06F16/2455 ,  H04L41/0896 ,  H04L67/1008 ,  H04L41/5025 ,  G06F9/50 ,  H04L43/0817

Abstract: A method for a multi-cluster warehouse includes allocating processing units as part of a data warehouse. The processing units access data within one or more databases in cloud storage resources. The method also includes providing one or more queries to each processing unit within the data warehouse. In response to the queries, each processing unit performs database operations on a particular portion of a database table. The method also includes monitoring a workload of the processing units to determine that a processing capacity of the processing units has reached a threshold processing capacity. The method also includes dynamically adding additional processing units to and removing processing units from the data warehouse as needed based on a workload of the processing units.

公开(公告)号：US20220391390A1

公开(公告)日：2022-12-08

申请号：US17879615

申请日：2022-08-02

Applicant: SNOWFLAKE INC.

Inventor： Thierry Cruanes ,  Florian Andreas Funke ,  Guangyan Hu ,  Jiaqi Yan

IPC: G06F16/2453 ,  G06F16/2455

Abstract: Joining data using a disjunctive operator is described. An example computer-implemented method can include generating a query plan for a query, wherein there is a join operator expression for each of a plurality of disjunctive predicates and each join operator expression includes at least a conjunctive predicate and a disjunctive operator. The method may also include generating a bloom filter for each of the plurality of disjunctive operators. The method may further include evaluating each of the plurality of join operator expressions using a corresponding one of the plurality of disjunctive operators and bloom filter for each of the plurality of disjunctive predicates to generate a result set.

公开(公告)号：US11144550B2

公开(公告)日：2021-10-12

申请号：US16857817

申请日：2020-04-24

Applicant: Snowflake Inc.

Inventor： Bowei Chen ,  Thierry Cruanes ,  Florian Andreas Funke ,  Allison Waingold Lee ,  Jiaqi Yan

IPC: G06F16/2453 ,  G06F16/2455 ,  G06F16/22

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and a join operation, the join operation including a build side and a probe side. The subject technology inserts an aggregation operator below the probe side of the join operation. The subject technology causes the build side of the join operation to generate a hash table. The subject technology causes the build side of the join operation to generate a bloom filter based at least in part on the hash table and provide information, corresponding to properties of the build side, to a bloom filter. Based at least in part on the information, the subject technology determines at least one property of the join operation to determine whether to switch the aggregation operator to a pass through mode.

公开(公告)号：US20210200769A1

公开(公告)日：2021-07-01

申请号：US17249794

申请日：2021-03-12

Applicant: Snowflake Inc.

Inventor： Florian Andreas Funke ,  Thierry Cruanes ,  Benoit Dageville ,  Marcin Zukowski

IPC: G06F16/2453 ,  G06F16/22 ,  G06F16/2455

Abstract: Systems, methods, and devices, for managing data skew during a join operation are disclosed. A method includes computing a hash value for a join operation and detecting data skew on a probe side of the join operation at a runtime of the join operation using a lightweight sketch data structure. The method includes identifying a frequent probe-side join key on the probe side of the join operation during a probe phase of the join operation. The method includes identifying a frequent build-side row having a build-side join key corresponding with the frequent probe-side join key. The method includes asynchronously distributing the frequent build-side row to one or more remote servers.

公开(公告)号：US20210173839A1

公开(公告)日：2021-06-10

申请号：US17180323

申请日：2021-02-19

Applicant: Snowflake Inc.

Inventor： Bowei Chen ,  Thierry Cruanes ,  Florian Andreas Funke ,  Allison Waingold Lee ,  Jiaqi Yan

IPC: G06F16/2453

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.