A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning
Summary: Controls all per-query tunable parameters in Spark’s Adaptive Query Execution via a hybrid compile-time/runtime, multi-granularity scheme to handle diverse, correlated parameters. Poses tuning as multi-objective (latency vs cost), with models/solvers achieving sub-second solves and much better latency/cost tradeoffs (63–65% latency reduction vs 18–25% for prior MOO methods) and superior adaptability to user preferences. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Chenghao Lyu (University of Massachusetts Amherst)
- 2. Qi Fan (Ecole Polytechnique)
- 3. Philippe Guyard (Ecole Polytechnique)
- 4. Yanlei Diao (Ecole Polytechnique; University of Massachusetts Amherst)
BibTeX Citation
@article{lyu_vldb24,
title = {{A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning}},
author = {Lyu, Chenghao and Fan, Qi and Guyard, Philippe and Diao, Yanlei},
journal = {PVLDB},
series = {{VLDB} '24},
volume = {17},
number = {11},
pages = {3565--3579},
doi = {10.14778/3681954.3682021},
url = {https://doi.org/10.14778/3681954.3682021},
year = {2024}
}Incoming Citations (Sorted by Pagerank)
Showing 4 of 4 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 9,052 | Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment | 2025 | SIGMOD | 5.215088e-05 |
| 9,995 | Graph Transformers for Query Plan Representation: Potentials and Challenges | 2025 | VLDB | 5.0845476e-05 |
| 10,800 | Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink | 2026 | VLDB | 4.9605343e-05 |
| 10,989 | ScaleSense: Cost-Intelligent Scaling Framework via Learned Resource Estimation in Alibaba AnalyticDB | 2026 | VLDB | 4.9605343e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 39 of 39 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 8,362 | SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft | 2021 | VLDB |
| 2 | 4,088 | Dynamically Optimizing Queries over Large Scale Data Platforms | 2014 | SIGMOD |
| 3 | 3,050 | Configuration-Parametric Query Optimization for Physical Design Tuning | 2008 | SIGMOD |
| 4 | 8,539 | New Query Optimization Techniques in the Spark Engine of Azure Synapse | 2022 | VLDB |
| 5 | 9,656 | Parallelizing Query Optimization on Shared-Nothing Architectures | 2016 | VLDB |
| 6 | 11,501 | LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison | 2025 | VLDB |
| 7 | 5,979 | Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems | 2019 | VLDB |
| 8 | 9,356 | Dynamic Speculative Optimizations for SQL Compilation in Apache Spark | 2020 | VLDB |
| 9 | 10,957 | MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning | 2026 | VLDB |
| 10 | 6,464 | Towards General and Efficient Online Tuning for Spark | 2023 | VLDB |