KEA: Tuning an Exabyte-Scale Data Infrastructure

Summary: KEA automates tuning of exabyte-scale data infra with ML models from telemetry, using observational tuning and cautious production flighting. First study addressing exabyte-scale data-management tuning, with potential tens of millions in annual savings. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6320
Venue: SIGMOD
Year: 2021
Pagerank: 5.7309848e-05
Overall Rank: 6,946 | 52.35%
DOI: 10.1145/3448016.3457569

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Subru Krishnan (Microsoft)
3. Konstantinos Karanasos (Microsoft)
4. Isha Tarte (Microsoft)
5. Conor Power (Microsoft)
6. Abhishek Modi (Microsoft)
7. Manoj Kumar (Microsoft)
8. Deli Zhang (Microsoft)
9. Kartheek Muthyala (Microsoft)
10. Nick Jurgens (Microsoft)
11. Sarvesh Sakalanaga (Salesforce)
12. Sudhir Darbha (Microsoft)
13. Minu Iyer (Microsoft)
14. Ankita Agarwal (Microsoft)
15. Carlo Curino (Microsoft)

BibTeX Citation

@inproceedings{zhu_sigmod21,
        title = {{KEA: Tuning an Exabyte-Scale Data Infrastructure}},
        author = {Zhu, Yiwen and Krishnan, Subru and Karanasos, Konstantinos and Tarte, Isha and Power, Conor and Modi, Abhishek and Kumar, Manoj and Zhang, Deli and Muthyala, Kartheek and Jurgens, Nick and Sakalanaga, Sarvesh and Darbha, Sudhir and Iyer, Minu and Agarwal, Ankita and Curino, Carlo},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457569},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457569},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 14 of 14 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,398	Real-time Workload Pattern Analysis for Large-scale Cloud Databases	2023	VLDB	6.7248611e-05
6,121	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.9688569e-05
6,177	Doppler: Automated SKU Recommendation in Migrating SQL Workloads to the Cloud	2022	VLDB	5.9496208e-05
6,344	Towards General and Efficient Online Tuning for Spark	2023	VLDB	5.9060457e-05
7,661	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.5736026e-05
7,762	Runtime Variation in Big Data Analytics	2023	SIGMOD	5.5501898e-05
8,193	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.4696038e-05
8,467	LST-Bench: Benchmarking Log-Structured Tables in the Cloud	2024	SIGMOD	5.4187125e-05
8,798	GEqO: ML-Accelerated Semantic Equivalence Detection	2023	SIGMOD	5.3698781e-05
8,861	Optimizing the cloud? Don't train models. Build oracles!	2024	CIDR	5.355716e-05
9,008	Making Data Clouds Smarter at Keebo: Automated Warehouse Optimization using Data Learning	2023	SIGMOD	5.3335632e-05
9,257	MLOS in Action: Bridging the Gap Between Experimentation and Auto-Tuning in the Cloud	2024	VLDB	5.2972217e-05
11,180	Lorentz: Learned SKU Recommendation Using Profile Data (DMDS)	2024	SIGMOD	5.093636e-05
11,222	Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service	2024	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00051174276
32	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00050111008
86	Automatic Database Management System Tuning Through Large-scale Machine Learning	2017	SIGMOD	0.00035316107
334	An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning	2019	SIGMOD	0.00020875082
735	Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs	2011	VLDB	0.00014522606
923	Starfish: A Self-tuning System for Big Data Analytics	2011	CIDR	0.00013189886
2,477	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.5239378e-05
3,466	Advanced Partitioning Techniques for Massively Distributed Computation	2012	SIGMOD	7.3909785e-05
3,931	Take me to your leader! Online Optimization of Distributed Storage Configurations	2015	VLDB	7.0093991e-05
3,964	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.9855158e-05
4,400	Recurring Job Optimization in Scope	2012	SIGMOD	6.7239302e-05
4,742	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	6.5269203e-05
6,822	JetScope: Reliable and Interactive Analytics at Cloud Scale	2015	VLDB	5.7627651e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,008	Making Data Clouds Smarter at Keebo: Automated Warehouse Optimization using Data Learning	2023	SIGMOD
2	6,121	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB
3	3,343	LlamaTune: Sample-Efficient DBMS Configuration Tuning	2022	VLDB
4	5,978	From Auto-tuning One Size Fits All to Self-designed and Learned Data-intensive Systems	2019	SIGMOD
5	5,865	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB
6	7,762	Runtime Variation in Big Data Analytics	2023	SIGMOD
7	4,742	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB
8	7,619	AutoToken: Predicting Peak Parallelism for Big Data Analytics at Microsoft	2020	VLDB
9	2,822	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
10	5,059	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD