Back to papers
Towards General and Efficient Online Tuning for Spark
Summary: General BO-based Spark tuner with a unified multi-objective/constraint formulation that performs online safe configuration search during real periodic job runs to eliminate offline evaluation overhead. Uses adaptive sub-space generation, approximate gradient descent, and meta-learning to accelerate search; deployed in production at Tencent, saving ~57% memory and ~35% CPU on 25K tasks within 20 iterations.
(summarized by gpt-5-mini on Feb 09 2026)
- Paper ID
- 13189
- Venue
- VLDB
- Year
- 2023
- Pagerank
- 5.0373773e-05
- Overall Rank
- 6,489 | 54.91%
- DOI
-
10.14778/3611540.3611548
Incoming Non-self Citations Over Time
Incoming Citations (Sorted by Pagerank)
Showing 6 of 6 citing papers.
Outgoing Citations (Sorted by Pagerank)
Showing 22 of 22 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank |
Cited Paper |
Year |
Venue |
Pagerank |
| 66 |
Spark SQL: Relational Data Processing in Spark |
2015 |
SIGMOD |
0.00061707583 |
| 183 |
Automatic Database Management System Tuning Through Large-scale Machine Learning |
2017 |
SIGMOD |
0.00036859633 |
| 287 |
Storm @Twitter |
2014 |
SIGMOD |
0.00028917909 |
| 423 |
Tuning Database Configuration Parameters with iTuned |
2009 |
VLDB |
0.00023628474 |
| 510 |
An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning |
2019 |
SIGMOD |
0.00021420477 |
| 539 |
Shark: SQL and Rich Analytics at Scale |
2013 |
SIGMOD |
0.00020615453 |
| 704 |
Query-based Workload Forecasting for Self-Driving Database Management Systems |
2018 |
SIGMOD |
0.00017785557 |
| 779 |
QTune: A Query-Aware Database Tuning System with Deep Reinforcement Learning |
2019 |
VLDB |
0.00016719473 |
| 822 |
Twitter Heron: Stream Processing at Scale |
2015 |
SIGMOD |
0.00016226851 |
| 1,048 |
Starfish: A Self-tuning System for Big Data Analytics |
2011 |
CIDR |
0.00014442178 |
| 1,892 |
Black or White? How to Develop an AutoTuner for Memory-based Analytics |
2020 |
SIGMOD |
0.00010176219 |
| 2,339 |
Samza: Stateful Scalable Stream Processing at LinkedIn |
2017 |
VLDB |
9.0029559e-05 |
| 2,845 |
VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition |
2021 |
VLDB |
8.0301674e-05 |
| 3,655 |
Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation |
2022 |
VLDB |
6.8723042e-05 |
| 3,905 |
A Demonstration of the OtterTune Automatic Database Management System Tuning Service |
2018 |
VLDB |
6.6430079e-05 |
| 3,995 |
ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases |
2021 |
SIGMOD |
6.5475871e-05 |
| 4,377 |
HUNTER: An Online Cloud Database Hybrid Tuning System for Personalized Requirements |
2022 |
SIGMOD |
6.2331947e-05 |
| 4,799 |
Towards Dynamic and Safe Configuration Tuning for Cloud Databases |
2022 |
SIGMOD |
5.9082876e-05 |
| 5,318 |
LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications |
2022 |
SIGMOD |
5.5685434e-05 |
| 6,376 |
A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning |
2023 |
SIGMOD |
5.0861082e-05 |
| 7,099 |
KEA: Tuning an Exabyte-Scale Data Infrastructure |
2021 |
SIGMOD |
4.8263529e-05 |
| 9,360 |
Efficient Big Data Processing in Hadoop MapReduce |
2012 |
VLDB |
4.3476444e-05 |
Semantically Similar Papers
| Overall Rank |
Paper |
Year |
Venue |
Pagerank |
| 3,207 |
Big Data Analytics with Datalog Queries on Spark |
2016 |
SIGMOD |
7.3847098e-05 |
| 9,196 |
Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale |
2022 |
VLDB |
4.3723457e-05 |
| 8,196 |
SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft |
2021 |
VLDB |
4.5568952e-05 |
| 9,643 |
Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment |
2025 |
SIGMOD |
4.3067693e-05 |
| 9,732 |
ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems |
2023 |
VLDB |
4.2901665e-05 |
| 8,454 |
Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance |
2024 |
VLDB |
4.5022073e-05 |
| 5,318 |
LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications |
2022 |
SIGMOD |
5.5685434e-05 |
| 4,799 |
Towards Dynamic and Safe Configuration Tuning for Cloud Databases |
2022 |
SIGMOD |
5.9082876e-05 |
| 6,265 |
Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems |
2019 |
VLDB |
5.1294788e-05 |
| 8,585 |
A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning |
2024 |
VLDB |
4.4856045e-05 |