SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft

Summary: SparkCruise injects a workload-driven feedback loop into the Spark SQL optimizer to optimize large workloads without accessing user data. Analysis of production Spark SQL workloads vs. TPC-DS demonstrates online learning and a computation-reuse optimization. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hd3243efe39024150
Venue: VLDB
Year: 2021
Pagerank: 5.3485189e-05
Overall Rank: 8,349 | 43.89%
DOI: 10.14778/3476311.3476388
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Abhishek Roy (Microsoft)
2. Alekh Jindal (Microsoft)
3. Priyanka Gomatam (Microsoft)
4. Xiating Ouyang (University of Wisconsin)
5. Ashit Gosalia (Microsoft)
6. Nishkam Ravi (Microsoft)
7. Swinky Mann (Microsoft)
8. Prakhar Jain (Databricks)

BibTeX Citation

@article{roy_vldb21,
        title = {{SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft}},
        author = {Roy, Abhishek and Jindal, Alekh and Gomatam, Priyanka and Ouyang, Xiating and Gosalia, Ashit and Ravi, Nishkam and Mann, Swinky and Jain, Prakhar},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {12},
        pages = {3122--3134},
        doi = {10.14778/3476311.3476388},
        url = {https://doi.org/10.14778/3476311.3476388},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,948	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	6.9051584e-05
6,199	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.850616e-05
8,357	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3463255e-05
9,031	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.2323895e-05
11,720	QaaD (Query-as-a-Data): Scalable Execution of Massive Number of Small Queries in Spark	2023	SIGMOD	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055384955
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050475202
296	Generic Schema Matching with Cupid	2001	VLDB	0.00021857869
432	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00018331051
555	SageDB: A Learned Database System	2019	CIDR	0.0001650754
1,432	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	0.00010676754
1,747	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.7303647e-05
1,797	IDEBench: A Benchmark for Interactive Data Exploration	2020	SIGMOD	9.6143465e-05
2,833	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9539771e-05
3,462	DIAMetrics: Benchmarking Query Engines at Scale	2020	VLDB	7.280037e-05
3,544	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2108612e-05
5,110	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD	6.2678118e-05
9,999	SparkCruise: Handsfree Computation Reuse in Spark	2019	VLDB	5.0964027e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,031	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD
2	10,896	MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning	2026	VLDB
3	8,608	Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance	2024	VLDB
4	3,455	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
5	9,330	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB
6	6,453	Towards General and Efficient Online Tuning for Spark	2023	VLDB
7	4,820	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB
8	8,542	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB
9	8,009	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
10	9,999	SparkCruise: Handsfree Computation Reuse in Spark	2019	VLDB