Runtime Variation in Big Data Analytics

Summary: Two-step predictor for runtime distribution: shape features plus a classifier with >96% accuracy. First large-scale study predicting enterprise analytics runtime categories; enables what-if analyses on allocation and scheduling. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6632
Venue: SIGMOD
Year: 2023
Pagerank: 5.5501898e-05
Overall Rank: 7,762 | 46.75%
DOI: 10.1145/3588921

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Rathijit Sen (Microsoft)
3. Robert Horton (Microsoft)
4. John Mark Agosta (Microsoft)

BibTeX Citation

@inproceedings{zhu_sigmod23,
        title = {{Runtime Variation in Big Data Analytics}},
        author = {Zhu, Yiwen and Sen, Rathijit and Horton, Robert and Agosta, John Mark},
        series = {{SIGMOD} '23},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3588921},
        url = {https://dl.acm.org/doi/10.1145/3588921},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
7,819	InTime: Towards Performance Predictability In Byzantine Fault Tolerant Proof-of-Stake Consensus	2025	SIGMOD	5.5373997e-05
10,018	From Logs to Causal Inference: Diagnosing Large Systems	2025	VLDB	5.1757914e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00051174276
32	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00050111008
86	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00035316107
334	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.00020875082
895	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00013357681
1,879	Predictable Performance for Unpredictable Workloads	2009	VLDB	9.5613416e-05
2,477	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.5239378e-05
3,441	A Statistical Perspective on Discovering Functional Dependencies in Noisy Data	2020	SIGMOD	7.4138323e-05
3,466	Advanced Partitioning Techniques for Massively Distributed Computation	2012	SIGMOD	7.3909785e-05
3,964	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.9855158e-05
4,400	Recurring Job Optimization in Scope	2012	SIGMOD	6.7239302e-05
4,742	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	6.5269203e-05
4,816	A Top-Down Approach to Achieving Performance Predictability in Database Systems	2017	SIGMOD	6.4949878e-05
6,102	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.976708e-05
6,121	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.9688569e-05
6,822	JetScope: Reliable and Interactive Analytics at Cloud Scale	2015	VLDB	5.7627651e-05
6,946	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD	5.7309848e-05
7,619	AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft	2020	VLDB	5.5810604e-05
9,224	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	5.3035811e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	6,946	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD
2	11,829	Automated Performance Management for the Big Data Stack	2019	CIDR
3	5,865	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB
4	624	Performance Prediction for Concurrent Database Workloads	2011	SIGMOD
5	3,605	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD
6	5,287	PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics	2013	VLDB
7	2,822	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
8	9,640	Supporting Scalable Analytics with Latency Constraints	2015	VLDB
9	895	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB
10	4,742	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB