Back to papers
Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment
Summary: Rockhopper introduces a noise-resilient Centroid Learning optimizer for Spark configuration tuning in production. It uses benchmark-informed workload embeddings for context-aware transfer learning and achieves ~20% gains by tuning only three query-level configurations.
(summarized by gpt-5-nano on Feb 09 2026)
- Paper ID
- 7115
- Venue
- SIGMOD
- Year
- 2025
- Pagerank
- 4.3067693e-05
- Overall Rank
- 9,643 | 32.99%
- DOI
-
10.1145/3722212.3724451
Incoming Non-self Citations Over Time
Incoming Citations (Sorted by Pagerank)
Showing 1 of 1 citing papers.
Outgoing Citations (Sorted by Pagerank)
Showing 17 of 17 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank |
Cited Paper |
Year |
Venue |
Pagerank |
| 183 |
Automatic Database Management System Tuning Through Large-scale Machine Learning |
2017 |
SIGMOD |
0.00036859633 |
| 510 |
An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning |
2019 |
SIGMOD |
0.00021420477 |
| 3,655 |
Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation |
2022 |
VLDB |
6.8723042e-05 |
| 3,905 |
A Demonstration of the OtterTune Automatic Database Management System Tuning Service |
2018 |
VLDB |
6.6430079e-05 |
| 4,180 |
LlamaTune: Sample-Efficient DBMS Configuration Tuning |
2022 |
VLDB |
6.3725334e-05 |
| 4,799 |
Towards Dynamic and Safe Configuration Tuning for Cloud Databases |
2022 |
SIGMOD |
5.9082876e-05 |
| 5,318 |
LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications |
2022 |
SIGMOD |
5.5685434e-05 |
| 5,876 |
AutoExecutor: Predictive Parallelism for Spark SQL Queries |
2021 |
VLDB |
5.2886967e-05 |
| 5,941 |
Eraser: Eliminating Performance Regression on Learned Query Optimizer |
2024 |
VLDB |
5.2594013e-05 |
| 6,489 |
Towards General and Efficient Online Tuning for Spark |
2023 |
VLDB |
5.0373773e-05 |
| 8,003 |
The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions |
2024 |
VLDB |
4.6049527e-05 |
| 8,196 |
SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft |
2021 |
VLDB |
4.5568952e-05 |
| 8,378 |
Towards Building Autonomous Data Services on Azure |
2023 |
SIGMOD |
4.5275731e-05 |
| 8,454 |
Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance |
2024 |
VLDB |
4.5022073e-05 |
| 8,585 |
A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning |
2024 |
VLDB |
4.4856045e-05 |
| 9,138 |
Phoebe: A Learning-based Checkpoint Optimizer |
2021 |
VLDB |
4.3842765e-05 |
| 9,194 |
MLOS in Action: Bridging the Gap Between Experimentation and Auto-Tuning in the Cloud |
2024 |
VLDB |
4.3726269e-05 |
Semantically Similar Papers
| Overall Rank |
Paper |
Year |
Venue |
Pagerank |
| 10,259 |
Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink |
2026 |
VLDB |
4.1905499e-05 |
| 557 |
SystemML: Declarative Machine Learning on Spark |
2016 |
VLDB |
0.00020186115 |
| 8,454 |
Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance |
2024 |
VLDB |
4.5022073e-05 |
| 6,265 |
Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems |
2019 |
VLDB |
5.1294788e-05 |
| 5,318 |
LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications |
2022 |
SIGMOD |
5.5685434e-05 |
| 3,536 |
Scaling Spark in the Real World: Performance and Usability |
2015 |
VLDB |
6.9938207e-05 |
| 10,872 |
LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison |
2025 |
VLDB |
4.1905499e-05 |
| 8,196 |
SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft |
2021 |
VLDB |
4.5568952e-05 |
| 6,489 |
Towards General and Efficient Online Tuning for Spark |
2023 |
VLDB |
5.0373773e-05 |
| 8,585 |
A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning |
2024 |
VLDB |
4.4856045e-05 |