公开(公告)号：US20220114180A1

公开(公告)日：2022-04-14

申请号：US17561222

申请日：2021-12-23

Applicant: Snowflake Inc.

Inventor： Subramanian Muralidhar ,  Benoit Dageville ,  Thierry Cruanes ,  Nileema Shingte ,  Saurin Shah ,  Torsten Grabs ,  Istvan Cseri

IPC: G06F16/2455 ,  G06F16/22 ,  G06F16/23

Abstract: Disclosed herein are systems and methods for selective scanning of external partitions. In an embodiment, a database platform receives a query directed at least in part to an external table stored on an external data storage platform. The external table is partitioned into partitions corresponding to storage locations in the external data storage platform. The database platform prunes, using external-table metadata that is stored by the database platform and that maps the partitions of the external table to the storage locations in the external data storage platform, those partitions that do not potentially contain data that satisfies the query. The database platform identifies data that satisfies the query by scanning any one or more of the partitions of the external table that were not pruned, and responds to the query at least in part with the identified data that satisfies the query.

公开(公告)号：US11294868B2

公开(公告)日：2022-04-05

申请号：US17459334

申请日：2021-08-27

Applicant: SNOWFLAKE INC.

Inventor： Pui Kei Johnston Chu ,  Benoit Dageville ,  Shreyas Narendra Desai ,  German Alberto Gil Echeverri ,  Prasanna Krishnan ,  Vishnu Dutt Paladugu ,  Bowen Zhang

IPC: G06F16/182 ,  G06F16/11 ,  G06F9/54 ,  G06F16/17

Abstract: Provided herein are systems and methods for an efficient method of replicating share objects to remote deployments. For example, the method may comprise modifying a share object of a first account of a data exchange into a global object wherein the share object includes grant metadata indicating share grants to a set of objects of a database. The method may further comprise creating, in a second account of the data exchange, a local replica of the share object on the remote deployment based on the global object, wherein the second account is located in a remote deployment. The set of objects of the database may be replicated to a local database replica on the remote deployment and the share grants may be replicated to the local replica of the share object.

公开(公告)号：US11269919B2

公开(公告)日：2022-03-08

申请号：US16945402

申请日：2020-07-31

Applicant: Snowflake Inc.

Inventor： Thierry Cruanes ,  Benoit Dageville ,  Marcin Zukowski

IPC: G06F15/167 ,  G06F16/27 ,  G06F9/50 ,  G06F16/14 ,  G06F16/21 ,  G06F16/22 ,  G06F16/951 ,  G06F16/182 ,  G06F16/23 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/9535 ,  G06F16/2453 ,  G06F9/48 ,  H04L67/1095 ,  H04L67/568 ,  H04L67/1097

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

公开(公告)号：US20220067068A1

公开(公告)日：2022-03-03

申请号：US17524454

申请日：2021-11-11

Applicant: SNOWFLAKE INC.

Inventor： Benoit Dageville ,  Thierry Cruanes ,  Marcin Zukowski ,  Allison Waingold Lee ,  Philipp Thomas Unterbrunner

IPC: G06F16/27 ,  G06F9/50 ,  G06F16/14 ,  G06F16/21 ,  G06F16/22 ,  G06F16/951 ,  G06F16/182 ,  G06F16/23 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/9535 ,  G06F16/2453 ,  G06F9/48 ,  H04L29/08

Abstract: A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components. An example method includes: receiving a relational join query for a join operation associated with a first relation and a second relation; generating at least one build operator and at least one probe operator to perform build operations and probe operations, respectively, of the join operation; and managing a state of one or more communication links between the at least one build operator and the at least one probe operator based on a size of the second relation as determined by the at least one build operator and an estimated size of the first relation.

公开(公告)号：US11263234B2

公开(公告)日：2022-03-01

申请号：US17497176

申请日：2021-10-08

Applicant: SNOWFLAKE INC.

Inventor： Benoit Dageville ,  Thierry Cruanes ,  Marcin Zukowski

IPC: G06F16/27 ,  G06F16/182 ,  G06F16/23 ,  G06F9/50 ,  G06F16/14 ,  G06F16/21 ,  G06F16/22 ,  G06F16/951 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/9535 ,  G06F16/2453 ,  G06F9/48 ,  H04L29/08 ,  H04L67/1095 ,  H04L67/568 ,  H04L67/1097

Abstract: A method and apparatus managing a set of processors for a set of queries is described. In an exemplary embodiment, a device receives a set of queries for a data warehouse, the set of queries including one or more queries to be processed by the data warehouse. The device further provisions a set of processors from a first plurality of processors, where the set of processors to process the set of queries, and a set of storage resources to store data for the set of queries. In addition, the device monitors a utilization of the set of processors as the set of processors processes the set of queries. The device additionally updates a number of the processors in the set of processors provisioned based on the utilization/Furthermore, the device processes the set of queries using the updated set of processors.

公开(公告)号：US20220035814A1

公开(公告)日：2022-02-03

申请号：US17502685

申请日：2021-10-15

Applicant: Snowflake Inc.

Inventor： Florian Andreas Funke ,  Thierry Cruanes ,  Benoit Dageville ,  Marcin Zukowski

IPC: G06F16/2453 ,  G06F16/22 ,  G06F16/2455

Abstract: Systems, methods, and devices, for managing data skew during a join operation are disclosed. A method includes computing a hash value for a join operation and detecting data skew on a probe side of the join operation at a runtime of the join operation using a lightweight sketch data structure. The method includes identifying a frequent probe-side join key on the probe side of the join operation during a probe phase of the join operation. The method includes identifying a frequent build-side row having a build-side join key corresponding with the frequent probe-side join key. The method includes asynchronously distributing the frequent build-side row to one or more remote servers.

公开(公告)号：US20220027368A1

公开(公告)日：2022-01-27

申请号：US17498382

申请日：2021-10-11

Applicant: Snowflake Inc.

Inventor： Subramanian Muralidhar ,  Benoit Dageville ,  Thierry Cruanes ,  Nileema Shingte ,  Saurin Shah ,  Torsten Grabs ,  Istvan Cseri

IPC: G06F16/2453

Abstract: Disclosed herein are systems and methods for processing queries over external tables. In an embodiment, a database platform receives a query directed at least to data in an external table stored in a storage platform that is external to the database platform. The database platform uses metadata that summarizes the data in the external table to identify one or more partitions of the external table as potentially including data satisfying the query, and generates a query plan that includes a plurality of discrete subtasks that collectively include instructions to scan the identified one or more partitions of the external table for data satisfying the query. The database platform assigns, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform, and refreshes the metadata in response to a threshold number of modifications being made to the external table.

公开(公告)号：US20220019570A1

公开(公告)日：2022-01-20

申请号：US17491106

申请日：2021-09-30

Applicant: Snowflake Inc.

Inventor： Subramanian Muralidhar ,  Istvan Cseri ,  Torsten Grabs ,  Benoit Dageville

IPC: G06F16/23 ,  G06F16/27 ,  G06F16/245 ,  G06F16/18

Abstract: Systems, methods, and devices for transactional processing of change tracking data for a database are discussed. A method includes generating a micro-partition based on execution of a transaction on a table of a database, the micro-partition reflecting changes made to the table by the transaction. A change tracking entry is generated in response to the execution of the transaction. The change tracking entry includes an indication of one or more modifications made to the table by the transaction and an indication of the micro-partition generated based on the execution of the transaction. The change tracking entry is stored in the micro-partition as metadata. At least one existing micro-partition is removed from the table, responsive to storing the change tracking entry.

公开(公告)号：US11204943B2

公开(公告)日：2021-12-21

申请号：US17086328

申请日：2020-10-31

Applicant: Snowflake Inc.

Inventor： Thierry Cruanes ,  Benoit Dageville ,  Allison Waingold Lee

IPC: G06F15/16 ,  G06F16/27 ,  G06F9/50 ,  G06F16/14 ,  G06F16/21 ,  G06F16/22 ,  G06F16/951 ,  G06F16/182 ,  G06F16/23 ,  G06F16/2455 ,  G06F16/2458 ,  G06F16/9535 ,  G06F16/2453 ,  G06F9/48 ,  H04L29/08

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

公开(公告)号：US11194795B2

公开(公告)日：2021-12-07

申请号：US16385837

申请日：2019-04-16

Applicant: Snowflake Inc.

Inventor： Subramanian Muralidhar ,  Benoit Dageville ,  Thierry Cruanes ,  Nileema Shingte ,  Saurin Shah ,  Torsten Grabs ,  Istvan Cseri

IPC: G06F16/20 ,  G06F16/242 ,  G06F3/06 ,  G06F16/23 ,  G06F16/16 ,  G06F16/25 ,  G06F9/54 ,  G06F16/22

Abstract: Systems, methods, and devices for automated maintenance of external tables in database systems are disclosed. A method includes receiving, by a database platform, read access to content in an external data storage platform that is separate from the database platform. The method includes defining an external table based on the content in the external data storage platform. The method includes connecting the database platform to the external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating metadata for the external table, the metadata comprising information about data stored in the external table. The method includes receiving a notification that a modification has been made to the content in the external data storage platform, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes refreshing the metadata for the external table in response to the modification being made to the content in the external data storage platform.