DBScholar

Back to papers

SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft

Summary: SparkCruise injects a workload-driven feedback loop into the Spark SQL optimizer to optimize large workloads without accessing user data. Analysis of production Spark SQL workloads vs. TPC-DS demonstrates online learning and a computation-reuse optimization. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
12705
Venue
VLDB
Year
2021
Pagerank
5.4737932e-05
Overall Rank
8,175 | 43.92%
DOI
10.14778/3476311.3476388

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{roy_vldb21,
        title = {{SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft}},
        author = {Roy, Abhishek and Jindal, Alekh and Gomatam, Priyanka and Ouyang, Xiating and Gosalia, Ashit and Ravi, Nishkam and Mann, Swinky and Jain, Prakhar},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {12},
        pages = {3122--3134},
        doi = {10.14778/3476311.3476388},
        url = {https://doi.org/10.14778/3476311.3476388},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank Citing Paper Year Venue Pagerank
3,998 Deploying a Steered Query Optimizer in Production at Microsoft 2022 SIGMOD 6.9676473e-05
6,102 AutoExecutor: Predictive Parallelism for Spark SQL Queries 2021 VLDB 5.976708e-05
8,193 Towards Building Autonomous Data Services on Azure 2023 SIGMOD 5.4696038e-05
9,771 Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment 2025 SIGMOD 5.2209769e-05
11,399 QaaD (Query-as-a-Data): Scalable Execution of Massive Number of Small Queries in Spark 2023 SIGMOD 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
24 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00054865648
30 SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets 2008 VLDB 0.00051174276
297 Generic Schema Matching with Cupid 2001 VLDB 0.00022157284
425 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018704491
568 SageDB: A Learned Database System 2019 CIDR 0.0001641553
1,468 Towards a Learning Optimizer for Shared Clouds 2019 VLDB 0.00010686496
1,765 Selecting Subexpressions to Materialize at Datacenter Scale 2018 VLDB 9.8079546e-05
1,767 IDEBench: A Benchmark for Interactive Data Exploration 2020 SIGMOD 9.805856e-05
2,822 Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings 2020 SIGMOD 8.0898536e-05
3,460 DIAMetrics: Benchmarking Query Engines at Scale 2020 VLDB 7.3939262e-05
3,605 Computation Reuse in Analytics Job Service at Microsoft 2018 SIGMOD 7.2640711e-05
5,059 Steering Query Optimizers: A Practical Take on Big Data Workloads 2021 SIGMOD 6.3807509e-05
9,854 SparkCruise: Handsfree Computation Reuse in Spark 2019 VLDB 5.2091816e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 11,222 Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service 2024 VLDB
2 9,771 Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment 2025 SIGMOD
3 8,457 Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance 2024 VLDB
4 3,411 Scaling Spark in the Real World: Performance and Usability 2015 VLDB
5 9,167 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB
6 6,344 Towards General and Efficient Online Tuning for Spark 2023 VLDB
7 4,742 Continuous Cloud-Scale Query Optimization and Processing 2013 VLDB
8 8,439 New Query Optimization Techniques in the Spark Engine of Azure Synapse 2022 VLDB
9 8,615 A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning 2024 VLDB
10 9,854 SparkCruise: Handsfree Computation Reuse in Spark 2019 VLDB