Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings

Summary: Workload-driven cost models for big data queries, integrated into a Cascade-style optimizer to optimize plans and containers. In production, Cleo/SCOPE sees 2–3 orders higher accuracy and 20x correlation; ~70% plan changes cut latency and save resources. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 5798
Venue: SIGMOD
Year: 2020
Pagerank: 6.9017341e-05
Overall Rank: 3,623 | 74.83%
DOI: 10.1145/3318464.3380584

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 40 of 40 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,955	Magpie: Python at Speed and Scale using Cloud Backends	2021	CIDR	7.8188583e-05
3,167	QueryFormer: A Tree Transformer Model for Query Plan Representation	2022	VLDB	7.4561078e-05
3,819	Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction	2022	VLDB	6.7267885e-05
3,866	Cloudy with High Chance of DBMS: A 10-year Prediction for Enterprise-Grade ML	2020	CIDR	6.6795497e-05
4,687	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	5.9915268e-05
5,339	LEON: A New Framework for ML-Aided Query Optimization	2023	VLDB	5.5596755e-05
5,373	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	5.5410059e-05
5,994	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD	5.2367998e-05
6,278	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.1241654e-05
6,774	A Unified Transferable Model for ML-Enhanced DBMS	2022	CIDR	4.9253635e-05
6,860	Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data	2023	SIGMOD	4.9008421e-05
6,883	PilotScope: Steering Databases with Machine Learning Drivers	2024	VLDB	4.8918682e-05
7,465	Yannakakis+: Practical Acyclic Query Evaluation with Theoretical Guarantees	2025	SIGMOD	4.7186055e-05
7,564	Modeling Shifting Workloads for Learned Database Systems	2024	SIGMOD	4.7049893e-05
7,652	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	4.6831938e-05
7,685	AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft	2020	VLDB	4.6753414e-05
7,742	Rethinking Learned Cost Models: Why Start from Scratch?	2023	SIGMOD	4.6585812e-05
7,787	Cost-Intelligent Data Analytics in the Cloud	2024	CIDR	4.6468106e-05
7,809	Sibyl: Forecasting Time-Evolving Query Workloads	2024	SIGMOD	4.6415559e-05
8,043	DISTILL: Low-Overhead Data-Driven Techniques for Filtering and Costing Indexes for Scalable Index Tuning	2022	VLDB	4.5954398e-05
8,196	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5568952e-05
8,219	PerfGuard: Deploying ML-for-Systems without Performance Regressions, Almost!	2021	VLDB	4.551524e-05
8,378	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	4.5275731e-05
8,579	Towards Query Optimizer as a Service (QOaaS) in a Unified LakeHouse Ecosystem: Can One QO Rule Them All?	2025	CIDR	4.4877266e-05
8,714	Tiresias: Enabling Predictive Autonomous Storage and Indexing	2022	VLDB	4.457682e-05
8,834	ByteCard: Enhancing ByteDance’s Data Warehouse with Learned Cardinality Estimation	2024	SIGMOD	4.4351469e-05
8,961	T3: Accurate and Fast Performance Prediction for Relational Database Systems With Compiled Decision Trees	2025	SIGMOD	4.4171776e-05
9,138	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	4.3842765e-05
9,600	Optimizing Dataflow Systems for Scalable Interactive Visualization	2024	SIGMOD	4.3136057e-05
9,841	Machine Unlearning in Learned Databases: An Experimental Analysis	2024	SIGMOD	4.2685233e-05
9,931	Wii: Dynamic Budget Reallocation In Index Tuning	2024	SIGMOD	4.2469394e-05
10,125	Understanding and Detecting Query Performance Regression in Practical Index Tuning: [Experiments & Analysis]	2026	SIGMOD	4.1905499e-05
10,219	Practical Parameterized Query Optimization via Efficient Plan Reuse and List-wise Ranking	2026	SIGMOD	4.1905499e-05
10,271	OBELISK: Efficient Offline Query Planning with Bayesian Optimization-Informed Language Model Reasoning	2026	VLDB	4.1905499e-05
10,501	Intra-Query Runtime Elasticity for Cloud-Native Data Analysis	2025	SIGMOD	4.1905499e-05
10,552	Esc: An Early-Stopping Checker for Budget-aware Index Tuning	2025	VLDB	4.1905499e-05
10,635	Robust Plan Evaluation based on Approximate Probabilistic Machine Learning	2025	VLDB	4.1905499e-05
10,853	AXE: A Task Decomposition Approach to Learned LSM Tuning	2025	VLDB	4.1905499e-05
10,863	Graph Transformers for Query Plan Representation: Potentials and Challenges	2025	VLDB	4.1905499e-05
11,269	Anser: Adaptive Information Sharing Framework of AnalyticDB	2023	VLDB	4.1905499e-05

Outgoing Citations (Sorted by Pagerank)

Showing 24 of 24 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
22	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00084679526
71	How Good Are Query Optimizers, Really?	2016	VLDB	0.00059446482
101	The Case for Learned Index Structures	2018	SIGMOD	0.00049778866
181	LEO - DB2's LEarning Optimizer	2001	VLDB	0.00036970794
203	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00034868567
329	Neo: A Learned Query Optimizer	2019	VLDB	0.00027301488
796	SageDB: A Learned Database System	2019	CIDR	0.00016541749
950	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00015100872
1,017	Robust Estimation of Resource Consumption for SQL Queries using Statistical Techniques	2012	VLDB	0.00014627121
1,229	Toward a Progress Indicator for Database Queries	2004	SIGMOD	0.00013153898
1,239	Selectivity Estimation for Range Predicates using Lightweight Models	2019	VLDB	0.00013091459
1,509	Estimating Progress of Execution for SQL Queries	2004	SIGMOD	0.0001158945
1,666	HELIX: Holistic Optimization for Accelerating Iterative Machine Learning	2019	VLDB	0.00010955907
1,921	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	0.00010085899
2,080	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	9.5954034e-05
3,044	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	7.6624689e-05
3,903	Automated Verification of Query Equivalence Using Satisfiability Modulo Theories	2019	VLDB	6.6439695e-05
4,128	Advanced Join Strategies for Large-Scale Distributed Computation	2014	VLDB	6.4214449e-05
4,171	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	6.3800823e-05
4,772	Recurring Job Optimization in Scope	2012	SIGMOD	5.9280555e-05
5,309	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5714729e-05
7,386	Bubble Execution: Resource-aware Reliable Analytics at Cloud Scale	2018	VLDB	4.7398757e-05
9,270	Redoop Infrastructure for Recurring Big Data Queries	2014	VLDB	4.362604e-05
9,734	SparkCruise: Handsfree Computation Reuse in Spark	2019	VLDB	4.2901665e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
3,819	Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction	2022	VLDB	6.7267885e-05
4,944	A Cost Model for Clustered Object-Oriented Databases	1995	VLDB	5.8120276e-05
4,698	Schedule Optimization for Data Processing Flows on the Cloud	2011	SIGMOD	5.9835195e-05
10,844	Learned Cost Models for Query Optimization: From Batch to Streaming Systems	2025	VLDB	4.1905499e-05
2,080	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	9.5954034e-05
6,687	How Good are Learned Cost Models, Really? Insights from Query Optimization Tasks	2025	SIGMOD	4.957987e-05
9,848	Saving Money for Analytical Workloads in the Cloud	2024	VLDB	4.2680295e-05
2,553	Towards Cost-Optimal Query Processing in the Cloud	2021	VLDB	8.5522099e-05
5,309	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5714729e-05
5,994	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD	5.2367998e-05