Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities
Summary: Analyzes 3,000 production ML pipelines at Google via provenance graphs and 450k trainings to characterize lifespan, topology, and complexity. Introduces model graphlets, a data model for repeated components, and shows optimization opportunities—pruning wasted computation can cut costs by ~50% without delaying deployment. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Doris Xin (University of California Berkeley)
- 2. Hui Miao (Google)
- 3. Aditya Parameswaran (University of California Berkeley)
- 4. Neoklis Polyzotis (Google)
BibTeX Citation
@inproceedings{xin_sigmod21,
title = {{Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities}},
author = {Xin, Doris and Miao, Hui and Parameswaran, Aditya and Polyzotis, Neoklis},
series = {{SIGMOD} '21},
booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
publisher = {Association for Computing Machinery},
doi = {10.1145/3448016.3457566},
url = {https://dl.acm.org/doi/10.1145/3448016.3457566},
year = {2021}
}Incoming Citations (Sorted by Pagerank)
Showing 12 of 12 citing papers.
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 14 of 14 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 2,018 | tf.data: A Machine Learning Data Processing Framework | 2021 | VLDB |
| 2 | 3,541 | Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines | 2022 | SIGMOD |
| 3 | 11,824 | Leveraging Organizational Resources to Adapt Models to New Data Modalities | 2020 | VLDB |
| 4 | 5,699 | Optimizing Machine Learning Workloads in Collaborative Environments | 2020 | SIGMOD |
| 5 | 11,516 | Data Management Opportunities for Foundation Models | 2022 | CIDR |
| 6 | 11,512 | Towards Observability for Machine Learning Pipelines | 2022 | CIDR |
| 7 | 8,050 | Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science | 2021 | VLDB |
| 8 | 1,147 | Data Management Challenges in Production Machine Learning | 2017 | SIGMOD |
| 9 | 9,245 | Towards Observability for Production Machine Learning Pipelines | 2022 | VLDB |
| 10 | 6,309 | Materialization and Reuse Optimizations for Production Data Science Pipelines | 2022 | SIGMOD |