Towards General and Efficient Online Tuning for Spark
Summary: General BO-based Spark tuner with a unified multi-objective/constraint formulation that performs online safe configuration search during real periodic job runs to eliminate offline evaluation overhead. Uses adaptive sub-space generation, approximate gradient descent, and meta-learning to accelerate search; deployed in production at Tencent, saving ~57% memory and ~35% CPU on 25K tasks within 20 iterations. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Yang Li (Tencent)
- 2. Huaijun Jiang (Peking University; Tencent)
- 3. Yu Shen (Peking University)
- 4. Yide Fang (Tencent)
- 5. Xiaofeng Yang (Tencent)
- 6. Danqing Huang (Tencent)
- 7. Xinyi Zhang (Peking University)
- 8. Wentao Zhang (Mila)
- 9. Ce Zhang (ETH Zurich)
- 10. Peng Chen (Tencent)
- 11. Bin Cui (Peking University)
BibTeX Citation
@article{li_vldb23,
title = {{Towards General and Efficient Online Tuning for Spark}},
author = {Li, Yang and Jiang, Huaijun and Shen, Yu and Fang, Yide and Yang, Xiaofeng and Huang, Danqing and Zhang, Xinyi and Zhang, Wentao and Zhang, Ce and Chen, Peng and Cui, Bin},
journal = {PVLDB},
series = {{VLDB} '23},
volume = {16},
number = {12},
pages = {3570--3583},
doi = {10.14778/3611540.3611548},
url = {https://doi.org/10.14778/3611540.3611548},
year = {2023}
}Incoming Citations (Sorted by Pagerank)
Showing 6 of 6 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 5,171 | An Efficient Transfer Learning Based Configuration Adviser for Database Tuning | 2024 | VLDB | 6.3347618e-05 |
| 8,457 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB | 5.4217837e-05 |
| 8,615 | A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning | 2024 | VLDB | 5.4005602e-05 |
| 9,771 | Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment | 2025 | SIGMOD | 5.2209769e-05 |
| 10,547 | Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink | 2026 | VLDB | 5.093636e-05 |
| 11,083 | Graph Transformers for Query Plan Representation: Potentials and Challenges | 2025 | VLDB | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 22 of 22 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 2,594 | Big Data Analytics with Datalog Queries on Spark | 2016 | SIGMOD |
| 2 | 9,290 | Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale | 2022 | VLDB |
| 3 | 8,175 | SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft | 2021 | VLDB |
| 4 | 9,771 | Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment | 2025 | SIGMOD |
| 5 | 9,898 | ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems | 2023 | VLDB |
| 6 | 8,457 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB |
| 7 | 4,854 | LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications | 2022 | SIGMOD |
| 8 | 4,011 | Towards Dynamic and Safe Configuration Tuning for Cloud Databases | 2022 | SIGMOD |
| 9 | 5,865 | Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems | 2019 | VLDB |
| 10 | 8,615 | A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning | 2024 | VLDB |