Database Paper Browser

Back to papers

Dynamically Optimizing Queries over Large Scale Data Platforms

Summary: Dynamic optimization of queries on large-scale data platforms, handling opaque UDFs and cross-relational correlations. Pilot runs estimate selectivities to seed a cost-based plan, after which plans evolve during execution to yield up to 2x (Jaql) and 4x (Hive) gains over hand-written left-deep baselines. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
4903
Venue
SIGMOD
Year
2014
Pagerank
5.7543101e-05
Overall Rank
5,012 | 65.17%
DOI
10.1145/2588555.2610531

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 11 of 11 citing papers.

Rank Citing Paper Year Venue Pagerank
1,756 Sampling-Based Query Re-Optimization 2016 SIGMOD 0.00010659753
2,080 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 9.5954034e-05
2,154 SkinnerDB: Regret-Bounded Query Evaluation via Reinforcement Learning 2018 VLDB 9.4176683e-05
2,222 SkinnerDB: Regret-Bounded Query Evaluation via Reinforcement Learning 2019 SIGMOD 9.2598438e-05
2,929 WANalytics: Analytics for a Geo-Distributed Data-Intensive World 2015 CIDR 7.8792902e-05
4,171 Computation Reuse in Analytics Job Service at Microsoft 2018 SIGMOD 6.3800823e-05
4,807 Resource Elasticity for Large-Scale Machine Learning 2015 SIGMOD 5.9045148e-05
8,162 MONSOON: Multi-Step Optimization and Execution of Queries with Partially Obscured Predicates 2020 SIGMOD 4.5686903e-05
9,122 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB 4.3877539e-05
9,123 On-Demand State Separation for Cloud Data Warehousing 2022 VLDB 4.3875153e-05
9,705 Distributed Numerical and Machine Learning Computations via Two-Phase Execution of Aggregated Join Trees 2021 VLDB 4.2951758e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 22 of 22 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
1 Access Path Selection in a Relational Database Management System 1979 SIGMOD 0.0040465394
42 A Comparison of Approaches to Large-Scale Data Analysis 2009 SIGMOD 0.00073570328
70 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00059744625
100 On the Propagation of Errors in the Size of Join Results 1991 SIGMOD 0.00050033475
107 Extensible/Rule Based Query Rewrite Optimization in Starburst 1992 SIGMOD 0.00048363446
221 Efficient Mid-Query Re-Optimization of Sub-Optimal Query Execution Plans 1998 SIGMOD 0.00033182072
224 CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies 2004 SIGMOD 0.00032743209
380 Towards Estimation Error Guarantees for Distinct Values 2000 PODS 0.00024943236
539 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00020615453
650 Robust Query Processing through Progressive Optimization 2004 SIGMOD 0.0001865144
737 On Synopses for Distinct-Value Estimation Under Multiset Operations 2007 SIGMOD 0.00017377393
776 Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience 2009 VLDB 0.00016765694
962 A Comparison of Join Algorithms for Log Processing in MapReduce 2010 SIGMOD 0.00015003834
1,265 Jaql: A Scripting Language for Large Scale Semistructured Data Analysis 2011 VLDB 0.00012933537
1,268 Proactive Re-Optimization 2005 SIGMOD 0.00012914584
1,451 Online Aggregation for Large MapReduce Jobs 2011 VLDB 0.00011925842
1,728 BigBench: Towards an Industry Standard Benchmark for Big Data Analytics 2013 SIGMOD 0.00010731422
1,790 Effective Use of Block-Level Sampling in Statistics Estimation 2004 SIGMOD 0.00010529479
2,616 Opening the Black Boxes in Data Flow Optimization 2012 VLDB 8.4457819e-05
2,754 Stubby: A Transformation-based Optimizer for MapReduce Workflows 2012 VLDB 8.1720428e-05
5,309 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB 5.5714729e-05
8,168 Progressive Optimization in a Shared-Nothing Parallel Database 2007 SIGMOD 4.5674166e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
3,729 Cost-based or Learning-based? A Hybrid Query Optimizer for Query Plan Selection 2022 VLDB 6.8078013e-05
6,372 Optimization of Conjunctive Predicates for Main Memory Column Stores 2016 VLDB 5.0878306e-05
6,177 Exploiting Soft and Hard Correlations in Big Data Query Optimization 2016 VLDB 5.1646971e-05
5,737 Enabling Incremental Query Re-Optimization 2016 SIGMOD 5.3459609e-05
1,621 Adaptive Optimization of Very Large Join Queries 2018 SIGMOD 0.00011105663
1,064 Analyzing Plan Diagrams of Database Query Optimizers 2005 VLDB 0.00014348262
332 Accelerating Machine Learning Inference with Probabilistic Predicates 2018 SIGMOD 0.00027173479
9,312 Parallelizing Query Optimization on Shared-Nothing Architectures 2016 VLDB 4.353536e-05
342 Optimization of Dynamic Query Evaluation Plans 1994 SIGMOD 0.00026836424
2,867 Optimization of Queries with User-defined Predicates 1996 VLDB 7.9931252e-05