Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment

Summary: Rockhopper introduces a noise-resilient Centroid Learning optimizer for Spark configuration tuning in production. It uses benchmark-informed workload embeddings for context-aware transfer learning and achieves ~20% gains by tuning only three query-level configurations. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 7114
Venue: SIGMOD
Year: 2025
Pagerank: 4.1945683e-05
Overall Rank: 10,414 | 27.56%
DOI: 10.1145/3722212.3724451

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank

Outgoing Citations (Sorted by Pagerank)

Showing 17 of 17 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
183	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00036721403
514	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.0002124895
3,812	Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation	2022	VLDB	6.7373184e-05
3,914	A Demonstration of the OtterTune Automatic Database Management System Tuning Service	2018	VLDB	6.6339644e-05
4,380	LlamaTune: Sample-Efficient DBMS Configuration Tuning	2022	VLDB	6.2396606e-05
4,842	Towards Dynamic and Safe Configuration Tuning for Cloud Databases	2022	SIGMOD	5.8826802e-05
5,833	LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications	2022	SIGMOD	5.3106182e-05
5,952	Eraser: Eliminating Performance Regression on Learned Query Optimizer	2024	VLDB	5.2591691e-05
6,209	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.1565972e-05
6,871	Towards General and Efficient Online Tuning for Spark	2023	VLDB	4.8997004e-05
8,020	The Holon Approach for Simultaneously Tuning Multiple Components in a Self-Driving Database Management System with Machine Learning via Synthesized Proto-Actions	2024	VLDB	4.6040862e-05
8,197	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5607121e-05
8,416	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	4.5196199e-05
8,617	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4846425e-05
9,155	Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance	2024	VLDB	4.3849295e-05
9,190	MLOS in Action: Bridging the Gap Between Experimentation and Auto-Tuning in the Cloud	2024	VLDB	4.3768215e-05
9,194	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	4.3761777e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
10,259	Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink	2026	VLDB	4.1945683e-05
557	SystemML: Declarative Machine Learning on Spark	2016	VLDB	0.00020197988
9,155	Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance	2024	VLDB	4.3849295e-05
6,268	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB	5.133857e-05
5,833	LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications	2022	SIGMOD	5.3106182e-05
3,535	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9992495e-05
10,868	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB	4.1945683e-05
8,197	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5607121e-05
6,871	Towards General and Efficient Online Tuning for Spark	2023	VLDB	4.8997004e-05
8,617	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4846425e-05