Abstract:
Techniques are described herein for subquery removal given two set operation-based subqueries in a query, where one subquery contains the result of the other. The described optimization technique of subquery removal is enabled by join and set operation-based containment of the set operation-based subqueries where semantic equivalence can be established for a given pair of set operation-based subqueries when some table(s)—with associated join condition(s), correlation condition(s), and/or filter predicate(s)—in one subquery are not considered. Subquery removal reduces multiple access to the same table and multiple evaluations of the same join conditions required to evaluate the query. When a subquery is removed from a disjunction, this may lead to other optimizations such as subquery unnesting, e.g., when the original query configuration would not permit query unnesting and the rewritten query (with one or more removed subqueries) permits unnesting.
Abstract:
According to embodiments, a multi-node database management system allows consumer processes (“consumers”) implementing a portion of a distributed data-combination operation to independently send a STOP notification to corresponding producer processes (“producers”). Upon a given consumer determining that the consumer requires no further information from corresponding producers, the consumer sends a STOP notification to the producers. When a given consumer sends out a STOP notification, the producers drop any data destined for the given consumer and also stops preparing data for and sending rows to the given consumer. Furthermore, once the producers receive STOP notifications from all of the consumers corresponding to the producers, the producers stop the current sub plan execution immediately without requiring completion of the sub plan. Thus, embodiments significantly improve the query execution performance by avoiding scanning and distributing data that is not needed for execution of the distributed operation.
Abstract:
According to one aspect of the invention, for a database statement that specifies evaluating ranking or cumulative window functions, an execution strategy based on an extended data distribution key may be used for the database statement. In the execution strategy, each sort operator of multiple parallel processing sort operators computes locally evaluated results of a ranking or cumulative window function based on a subset of rows in all rows used to evaluate the database statement, and sends the first and last rows' locally evaluated results to a query coordinator. The query coordinator consolidates the locally evaluated results received from the multiple parallel processing sort operators and sends consolidated results to the sort operators based on their respective demographics. Each sort operator completes full evaluation of the ranking or cumulative window functions based at least in part on one or more of the consolidated results provided by the query coordinator.
Abstract:
Execution plans generated for multiple analytic queries incorporate two new kinds of plan operators, a partition creator and partition iterator. The partition creator and partition iterator operate as a pair. A partition creator operator creates partitions of rows and a partitioning descriptor describing the partitions created. A partition iterator iterates through the partitions based on the partitioning descriptor. For each partition, multiple analytic operators are executed serially, one after the other, on the same rows in the partition. According to an embodiment, partitioning is based on a common grouping or subgrouping of the multiple analytic functions or operators. Columns in the grouping or subgrouping may be ignored when executing each of the multiple analytic operators. Forming execution plans that include partition creator and partition iterator in this way is referred to herein as partitioning injection.
Abstract:
A method, apparatus, and system for dynamic parallel aggregation with hybrid batch flushing are provided. Record sources of an aggregation operator in a query execution plan may dynamically aggregate using the same aggregation operator. The dynamic aggregation creates a batch of aggregation records from an input source, which are then used to aggregate further records from the input source. If a record from the input source is not matched to an aggregation record in the batch, then the record is passed to the next operator. In this manner, records are aggregated ahead of time at a record source to reduce the number of records passed between operators, reducing the impact of network I/O between nodes of a parallel processing system. By adjusting the contents of the batch according to aggregation performance monitored during run-time, hybrid batch flushing can be implemented to adapt to changing data patterns and skewed values.
Abstract:
According to one aspect of the invention, for a database statement that specifies rollup operations, a data distribution key may be selected among a plurality of candidate keys. Numbers of distinct values of the candidate keys may be monitored with respect to a particular set of rows. Hash values may also be generated by column values in the candidate keys. The data distribution key may be determined based on results of monitoring the numbers of distinct values of the candidate keys as well as the frequencies of hash values computed based on column values of the candidate keys. Rollup operations may be shared between different stages of parallel executing processes and data may be distributed between the different stages of parallel executing processes based on the selected data distribution key.
Abstract:
According to one aspect of the invention, for a database statement that specifies evaluating reporting window functions, a computation-pushdown execution strategy may be used for the database statement. The computation-pushdown execution plan includes producer operators and consolidation operators. Each producer operator computes a respective partial aggregation for each reporting window function based on a subset of rows, and broadcasts the respective partial aggregation. Each consolidation operator fully aggregates all partial aggregations broadcasted from the producer operators. Alternatively, an extended-data-distribution-key execution plan may be used. Each producer operator sends rows based on hash keys to sort operators for computing partial aggregations for at least one reporting window function based on a subset of rows. Each consolidation operator receives and fully aggregates all partial aggregations broadcasted from the sort operators.
Abstract:
According to one aspect of the invention, for a database statement that specifies rollup operations, a data distribution key may be selected among a plurality of candidate keys. Numbers of distinct values of the candidate keys may be monitored with respect to a particular set of rows. Hash values may also be generated by column values in the candidate keys. The data distribution key may be determined based on results of monitoring the numbers of distinct values of the candidate keys as well as the frequencies of hash values computed based on column values of the candidate keys. Rollup operations may be shared between different stages of parallel executing processes and data may be distributed between the different stages of parallel executing processes based on the selected data distribution key.
Abstract:
Techniques herein generate a query plan that combines a global reporting aggregate calculation and an organizing operation. A method detects an organizing operation, a group aggregate function, and a global aggregate function within a database statement. The organizing operation specifies organizational activities such as grouping, joining, or sorting rows. The method generates an execution plan that specifies calculating all values in a single pass. For each row, the single pass applies the organizing operation and updates an access structure. The pass updates one of multiple cumulative group calculations based on the group aggregate function and updates a cumulative global calculation based on the global aggregate function. Each cumulative group calculation is associated with some of the access structure. Based on the access structure, result rows that satisfy the database statement are generated. Result rows contain a final result of each group calculation and a final result of the global calculation.
Abstract:
Techniques herein improve computational efficiency for parallel queries with run-time data pruning by using adaptive granule generation. In an embodiment, an execution plan is generated for a query to be executed by a plurality of slave processes, the execution plan comprising a plurality of plan operators. For a first plan operator of the plurality of plan operators, a first set of work granules is generated, and for a second plan operator of the plurality of plan operators, a second set of work granules is generated. A first subset of slave processes of the plurality of slave processes is assigned the first set of work granules. Based on the execution of the first set of work granules by the first subset of slave processes, a bloom filter is generated that specifies for which of said first set of work granules no output rows were generated. Based on the bloom filter, the second set of work granules is modified and the modified second set of work granules is assigned to a second subset of slave processes and executed.