A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

Summary: Controls all per-query tunable parameters in Spark’s Adaptive Query Execution via a hybrid compile-time/runtime, multi-granularity scheme to handle diverse, correlated parameters. Poses tuning as multi-objective (latency vs cost), with models/solvers achieving sub-second solves and much better latency/cost tradeoffs (63–65% latency reduction vs 18–25% for prior MOO methods) and superior adaptability to user preferences. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13564
Venue: VLDB
Year: 2024
Pagerank: 4.4846425e-05
Overall Rank: 8,617 | 40.06%
DOI: 10.14778/3681954.3682021

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
10,259	Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink	2026	VLDB	4.1945683e-05
10,414	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	4.1945683e-05
10,859	Graph Transformers for Query Plan Representation: Potentials and Challenges	2025	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 39 of 39 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061639801
70	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00059533166
183	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00036721403
514	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.0002124895
542	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00020595648
780	Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience	2009	VLDB	0.00016775082
782	QTune: A Query-Aware Database Tuning System with Deep Reinforcement Learning	2019	VLDB	0.00016729063
1,647	Parametric Query Optimization for Linear and Piecewise Linear Cost Functions	2002	VLDB	0.00011033757
1,684	Fuxi: a Fault-Tolerant Resource Management and Job Scheduling System at Internet Scale	2014	VLDB	0.0001091857
1,902	Black or White? How to Develop an AutoTuner for Memory-based Analytics	2020	SIGMOD	0.00010157713
2,364	Deep Learning Models for Selectivity Estimation of Multi-Attribute Queries	2020	SIGMOD	8.9554751e-05
2,568	Towards Cost-Optimal Query Processing in the Cloud	2021	VLDB	8.5239227e-05
2,659	Multi-Objective Parametric Query Optimization	2015	VLDB	8.3604734e-05
2,691	Greenplum: A Hybrid Database for Transactional and Analytical Workloads	2021	SIGMOD	8.2909126e-05
2,762	FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation	2021	VLDB	8.1585394e-05
2,783	Flow-Loss: Learning Cardinality Estimates That Matter	2021	VLDB	8.1293383e-05
3,216	WiSeDB: A Learning-based Workload Management Advisor for Cloud Databases	2016	VLDB	7.3601267e-05
3,499	Fauce: Fast and Accurate Deep Ensembles with Uncertainty for Cardinality Estimation	2021	VLDB	7.0376445e-05
3,522	ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases	2021	SIGMOD	7.0096727e-05
3,924	A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation	2021	SIGMOD	6.6271553e-05
4,543	FACE: A Normalizing Flow based Cardinality Estimator	2022	VLDB	6.1011198e-05
4,700	Schedule Optimization for Data Processing Flows on the Cloud	2011	SIGMOD	5.9882572e-05
4,842	Towards Dynamic and Safe Configuration Tuning for Cloud Databases	2022	SIGMOD	5.8826802e-05
4,874	Approximation Schemes for Many-Objective Query Optimization	2014	SIGMOD	5.8594632e-05
4,913	UDO: Universal Database Optimization using Reinforcement Learning	2021	VLDB	5.8316231e-05
5,075	An Incremental Anytime Algorithm for Multi-Objective Query Optimization	2015	SIGMOD	5.7172118e-05
5,368	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	5.5457532e-05
5,469	Learned Cardinality Estimation for Similarity Queries	2021	SIGMOD	5.4898192e-05
5,531	Presto: A Decade of SQL Analytics at Meta	2023	SIGMOD	5.4549499e-05
5,634	Intelligent Scaling in Amazon Redshift	2024	SIGMOD	5.4000904e-05
5,833	LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications	2022	SIGMOD	5.3106182e-05
6,368	Pre-training Summarization Models of Structured Datasets for Cardinality Estimation	2022	VLDB	5.0937722e-05
6,871	Towards General and Efficient Online Tuning for Spark	2023	VLDB	4.8997004e-05
7,358	Weighted Distinct Sampling: Cardinality Estimation for SPJ Queries	2021	SIGMOD	4.7529363e-05
8,576	PostCENN: PostgreSQL with Machine Learning Models for Cardinality Estimation	2021	VLDB	4.4927989e-05
9,066	Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases	2016	VLDB	4.4035481e-05
9,546	Trident: Task Scheduling over Tiered Storage Systems in Big Data Platforms	2021	VLDB	4.3259935e-05
9,547	Optimistic Recovery for Iterative Dataflows in Action	2015	SIGMOD	4.3259935e-05
9,736	UDAO: A Next-Generation Unified Data Analytics Optimizer	2019	VLDB	4.2942813e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
6,667	Leveraging Query Logs and Machine Learning for Parametric Query Optimization	2022	VLDB	4.9688874e-05
8,197	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5607121e-05
5,014	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	5.7586174e-05
3,284	Configuration-Parametric Query Optimization for Physical Design Tuning	2008	SIGMOD	7.2790444e-05
8,506	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB	4.4957661e-05
9,305	Parallelizing Query Optimization on Shared-Nothing Architectures	2016	VLDB	4.3577129e-05
10,868	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB	4.1945683e-05
6,268	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB	5.133857e-05
9,124	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB	4.391961e-05
6,871	Towards General and Efficient Online Tuning for Spark	2023	VLDB	4.8997004e-05