Towards General and Efficient Online Tuning for Spark
Summary: General BO-based Spark tuner with a unified multi-objective/constraint formulation that performs online safe configuration search during real periodic job runs to eliminate offline evaluation overhead. Uses adaptive sub-space generation, approximate gradient descent, and meta-learning to accelerate search; deployed in production at Tencent, saving ~57% memory and ~35% CPU on 25K tasks within 20 iterations. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Yang Li (Tencent)
- 2. Huaijun Jiang (Peking University; Tencent)
- 3. Yu Shen (Peking University)
- 4. Yide Fang (Tencent)
- 5. Xiaofeng Yang (Tencent)
- 6. Danqing Huang (Tencent)
- 7. Xinyi Zhang (Peking University)
- 8. Wentao Zhang (Mila)
- 9. Ce Zhang (ETH Zurich)
- 10. Peng Chen (Tencent)
- 11. Bin Cui (Peking University)
BibTeX Citation
@article{li_vldb23,
title = {{Towards General and Efficient Online Tuning for Spark}},
author = {Li, Yang and Jiang, Huaijun and Shen, Yu and Fang, Yide and Yang, Xiaofeng and Huang, Danqing and Zhang, Xinyi and Zhang, Wentao and Zhang, Ce and Chen, Peng and Cui, Bin},
journal = {PVLDB},
series = {{VLDB} '23},
volume = {16},
number = {12},
pages = {3570--3583},
doi = {10.14778/3611540.3611548},
url = {https://doi.org/10.14778/3611540.3611548},
year = {2023}
}Incoming Citations (Sorted by Pagerank)
Showing 7 of 7 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 5,304 | An Efficient Transfer Learning Based Configuration Adviser for Database Tuning | 2024 | VLDB | 6.1758143e-05 |
| 8,026 | A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning | 2024 | VLDB | 5.3884724e-05 |
| 8,626 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB | 5.2851279e-05 |
| 9,052 | Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment | 2025 | SIGMOD | 5.215088e-05 |
| 9,995 | Graph Transformers for Query Plan Representation: Potentials and Challenges | 2025 | VLDB | 5.0845476e-05 |
| 10,800 | Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink | 2026 | VLDB | 4.9605343e-05 |
| 10,957 | MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning | 2026 | VLDB | 4.9605343e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 22 of 22 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next