Runtime Variation in Big Data Analytics

Summary: Two-step predictor for runtime distribution: shape features plus a classifier with >96% accuracy. First large-scale study predicting enterprise analytics runtime categories; enables what-if analyses on allocation and scheduling. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h3321c7a84daf9095
Venue: SIGMOD
Year: 2023
Pagerank: 5.4051579e-05
Overall Rank: 7,944 | 46.76%
DOI: 10.1145/3588921

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Rathijit Sen (Microsoft)
3. Robert Horton (Microsoft)
4. John Mark Agosta (Microsoft)

BibTeX Citation

@inproceedings{zhu_sigmod23,
        title = {{Runtime Variation in Big Data Analytics}},
        author = {Zhu, Yiwen and Sen, Rathijit and Horton, Robert and Agosta, John Mark},
        series = {{SIGMOD} '23},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3588921},
        url = {https://dl.acm.org/doi/10.1145/3588921},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
8,003	InTime: Towards Performance Predictability In Byzantine Fault Tolerant Proof-of-Stake Consensus	2025	SIGMOD	5.392702e-05
10,250	From Logs to Causal Inference: Diagnosing Large Systems	2025	VLDB	5.0405429e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050364918
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049707539
74	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00037164008
309	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.0002143249
893	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00013199466
1,926	Predictable Performance for Unpredictable Workloads	2009	VLDB	9.3500398e-05
2,452	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4280184e-05
3,205	A Statistical Perspective on Discovering Functional Dependencies in Noisy Data	2020	SIGMOD	7.5221127e-05
3,453	Advanced Partitioning Techniques for Massively Distributed Computation	2012	SIGMOD	7.2720027e-05
3,907	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.9139522e-05
4,492	Recurring Job Optimization in Scope	2012	SIGMOD	6.5612113e-05
4,842	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	6.3734445e-05
4,932	A Top-Down Approach to Achieving Performance Predictability in Database Systems	2017	SIGMOD	6.3369302e-05
6,212	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.8312703e-05
6,260	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.815273e-05
6,981	JetScope: Reliable and Interactive Analytics at Cloud Scale	2015	VLDB	5.6127553e-05
7,108	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD	5.5817869e-05
7,764	AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft	2020	VLDB	5.4403231e-05
9,394	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	5.1676892e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	12,180	Automated Performance Management for the Big Data Stack	2019	CIDR
2	5,979	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB
3	10,826	A Resource-centric Analysis and Optimization of NoSQL Workloads using Distressed Resource Volume Metric	2026	VLDB
4	628	Performance Prediction for Concurrent Database Workloads	2011	SIGMOD
5	3,550	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD
6	5,394	PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics	2013	VLDB
7	2,828	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
8	9,857	Supporting Scalable Analytics with Latency Constraints	2015	VLDB
9	893	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB
10	4,842	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB