Runtime Variation in Big Data Analytics

Summary: Two-step predictor for runtime distribution: shape features plus a classifier with >96% accuracy. First large-scale study predicting enterprise analytics runtime categories; enables what-if analyses on allocation and scheduling. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6571
Venue: SIGMOD
Year: 2023
Pagerank: 4.6491879e-05
Overall Rank: 7,778 | 45.95%
DOI: 10.1145/3588921

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
7,849	InTime: Towards Performance Predictability In Byzantine Fault Tolerant Proof-of-Stake Consensus	2025	SIGMOD	4.6321834e-05
9,870	From Logs to Causal Inference: Diagnosing Large Systems	2025	VLDB	4.2626861e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
22	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00084679526
70	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00059744625
183	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00036859633
510	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.00021420477
950	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00015100872
2,372	Predictable Performance for Unpredictable Workloads	2009	VLDB	8.940791e-05
3,044	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	7.6624689e-05
4,068	Advanced Partitioning Techniques for Massively Distributed Computation	2012	SIGMOD	6.4748133e-05
4,129	A Statistical Perspective on Discovering Functional Dependencies in Noisy Data	2020	SIGMOD	6.4208557e-05
4,226	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.3382156e-05
4,772	Recurring Job Optimization in Scope	2012	SIGMOD	5.9280555e-05
5,309	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5714729e-05
5,518	A Top-Down Approach to Achieving Performance Predictability in Database Systems	2017	SIGMOD	5.468383e-05
5,876	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.2886967e-05
6,278	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.1241654e-05
7,079	JetScope: Reliable and Interactive Analytics at Cloud Scale	2015	VLDB	4.8353804e-05
7,099	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD	4.8263529e-05
7,685	AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft	2020	VLDB	4.6753414e-05
9,138	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	4.3842765e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
7,099	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD	4.8263529e-05
11,639	Automated Performance Management for the Big Data Stack	2019	CIDR	4.1905499e-05
6,265	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB	5.1294788e-05
716	Performance Prediction for Concurrent Database Workloads	2011	SIGMOD	0.00017623897
4,171	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	6.3800823e-05
5,699	PREDIcT: Towards Predicting the Runtime of Large Scale Iterative Analytics	2013	VLDB	5.3655096e-05
3,623	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	6.9017341e-05
9,505	Supporting Scalable Analytics with Latency Constraints	2015	VLDB	4.3300131e-05
950	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00015100872
5,309	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5714729e-05