Materialization and Reuse Optimizations for Production Data Science Pipelines
Summary: Proposes budgeted materialization to precompute and store pipeline artifacts, reducing redundant data processing in retraining ML pipelines. Introduces a DAG-based reuse planner to fuse pipelines and reuse artifacts, delivering up to 10x training speedups. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Behrouz Derakhshan (German National Research Center for Information Technology)
- 2. Alireza Rezaei Mahdiraji (Yara Digital Production)
- 3. Zoi Kaoudi (Technical University of Berlin)
- 4. Tilmann Rabl (Hasso Plattner Institute)
- 5. Volker Markl (German National Research Center for Information Technology; Technical University of Berlin)
BibTeX Citation
@inproceedings{derakhshan_sigmod22,
title = {{Materialization and Reuse Optimizations for Production Data Science Pipelines}},
author = {Derakhshan, Behrouz and Mahdiraji, Alireza Rezaei and Kaoudi, Zoi and Rabl, Tilmann and Markl, Volker},
series = {{SIGMOD} '22},
booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
publisher = {Association for Computing Machinery},
doi = {10.1145/3514221.3526186},
url = {https://dl.acm.org/doi/10.1145/3514221.3526186},
year = {2022}
}Incoming Citations (Sorted by Pagerank)
Showing 5 of 5 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 5,793 | Optimizing Data Pipelines for Machine Learning in Feature Stores | 2023 | VLDB | 5.9761004e-05 |
| 7,558 | Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines | 2023 | SIGMOD | 5.4788791e-05 |
| 9,099 | Scheduling Data Processing Pipelines for Incremental Training on MLP-based Recommendation Models | 2025 | SIGMOD | 5.208561e-05 |
| 11,002 | IMLane: Composable Framework for Efficient AI Function Execution in Database Engine | 2026 | VLDB | 4.9605343e-05 |
| 11,603 | Pasta: A Cost-Based Optimizer for Generating Pipelining Schedules for Dataflow DAGs | 2024 | SIGMOD | 4.9605343e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 18 of 18 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 1,151 | Data Management Challenges in Production Machine Learning | 2017 | SIGMOD |
| 2 | 3,598 | Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines | 2022 | SIGMOD |
| 3 | 2,062 | tf.data: A Machine Learning Data Processing Framework | 2021 | VLDB |
| 4 | 9,454 | Towards Observability for Production Machine Learning Pipelines | 2022 | VLDB |
| 5 | 5,793 | Optimizing Data Pipelines for Machine Learning in Feature Stores | 2023 | VLDB |
| 6 | 4,354 | LIMA: Fine-grained Lineage Tracing and Reuse in Machine Learning Systems | 2021 | SIGMOD |
| 7 | 7,558 | Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines | 2023 | SIGMOD |
| 8 | 6,164 | Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse | 2018 | VLDB |
| 9 | 5,808 | Optimizing Machine Learning Workloads in Collaborative Environments | 2020 | SIGMOD |
| 10 | 2,116 | Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities | 2021 | SIGMOD |