AutoExecutor: Predictive Parallelism for Spark SQL Queries
Summary: Predicts Spark SQL runtimes as a function of executor count to guide upfront resource sizing. AutoExecutor uses ML models to bound maximum parallelism, yielding cost-efficient performance on Azure Synapse. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Rathijit Sen (Microsoft)
- 2. Abhishek Roy (Microsoft)
- 3. Alekh Jindal (Microsoft)
- 4. Rui Fang (Microsoft)
- 5. Jeff Zheng (Microsoft)
- 6. Xiaolei Liu (Microsoft)
- 7. Ruiping Li (Microsoft)
BibTeX Citation
@article{sen_vldb21,
title = {{AutoExecutor: Predictive Parallelism for Spark SQL Queries}},
author = {Sen, Rathijit and Roy, Abhishek and Jindal, Alekh and Fang, Rui and Zheng, Jeff and Liu, Xiaolei and Li, Ruiping},
journal = {PVLDB},
series = {{VLDB} '21},
volume = {14},
number = {12},
pages = {2855--2858},
doi = {10.14778/3476311.3476362},
url = {https://doi.org/10.14778/3476311.3476362},
year = {2021}
}Incoming Citations (Sorted by Pagerank)
Showing 4 of 4 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6,177 | Doppler: Automated SKU Recommendation in Migrating SQL Workloads to the Cloud | 2022 | VLDB | 5.9496208e-05 |
| 7,762 | Runtime Variation in Big Data Analytics | 2023 | SIGMOD | 5.5501898e-05 |
| 8,457 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB | 5.4217837e-05 |
| 9,771 | Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment | 2025 | SIGMOD | 5.2209769e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 3 of 3 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 30 | SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets | 2008 | VLDB | 0.00051174276 |
| 7,619 | AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft | 2020 | VLDB | 5.5810604e-05 |
| 8,175 | SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft | 2021 | VLDB | 5.4737932e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 11,028 | Smart SPARQL Advisor: Guiding Users in Query Formulation with Performance Prediction | 2025 | VLDB |
| 2 | 434 | Query Optimization for Parallel Execution | 1992 | SIGMOD |
| 3 | 11,845 | Query-Driven Learning for Next Generation Predictive Modeling & Analytics | 2019 | SIGMOD |
| 4 | 11,361 | Predicting Query Execution time for JIT Compiled Database Engines | 2023 | CIDR |
| 5 | 3,482 | Towards Predicting Query Execution Time for Concurrent and Dynamic Database Workloads | 2013 | VLDB |
| 6 | 5,132 | Facilitating SQL Query Composition and Analysis | 2020 | SIGMOD |
| 7 | 8,615 | A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning | 2024 | VLDB |
| 8 | 8,439 | New Query Optimization Techniques in the Spark Engine of Azure Synapse | 2022 | VLDB |
| 9 | 11,222 | Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service | 2024 | VLDB |
| 10 | 7,619 | AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft | 2020 | VLDB |