Materialization and Reuse Optimizations for Production Data Science Pipelines

Summary: Proposes budgeted materialization to precompute and store pipeline artifacts, reducing redundant data processing in retraining ML pipelines. Introduces a DAG-based reuse planner to fuse pipelines and reuse artifacts, delivering up to 10x training speedups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6555
Venue: SIGMOD
Year: 2022
Pagerank: 5.9189554e-05
Overall Rank: 6,309 | 56.72%
DOI: 10.1145/3514221.3526186

Incoming Non-self Citations Over Time

Authors

1. Behrouz Derakhshan (German National Research Center for Information Technology)
2. Alireza Rezaei Mahdiraji (Yara Digital Production)
3. Zoi Kaoudi (Technical University of Berlin)
4. Tilmann Rabl (Hasso Plattner Institute)
5. Volker Markl (German National Research Center for Information Technology; Technical University of Berlin)

BibTeX Citation

@inproceedings{derakhshan_sigmod22,
        title = {{Materialization and Reuse Optimizations for Production Data Science Pipelines}},
        author = {Derakhshan, Behrouz and Mahdiraji, Alireza Rezaei and Kaoudi, Zoi and Rabl, Tilmann and Markl, Volker},
        series = {{SIGMOD} '22},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3514221.3526186},
        url = {https://dl.acm.org/doi/10.1145/3514221.3526186},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,704	Optimizing Data Pipelines for Machine Learning in Feature Stores	2023	VLDB	6.1146371e-05
7,395	Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines	2023	SIGMOD	5.6257796e-05
8,907	Scheduling Data Processing Pipelines for Incremental Training on MLP-based Recommendation Models	2025	SIGMOD	5.3483178e-05
11,207	Pasta: A Cost-Based Optimizer for Generating Pipelining Schedules for Dataflow DAGs	2024	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 18 of 18 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
82	Space-Efficient Online Computation of Quantile Summaries	2001	SIGMOD	0.00036378991
128	Efficient and Extensible Algorithms for Multi Query Optimization	2000	SIGMOD	0.0003072825
579	Incremental Knowledge Base Construction Using DeepDive	2015	VLDB	0.00016217563
640	Materialization Optimizations for Feature Selection Workloads	2014	SIGMOD	0.00015409494
803	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB	0.00013899943
822	Pipelining in Multi-Query Optimization	2001	PODS	0.00013807229
1,349	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011069122
1,354	On-the-Fly Sharing for Streamed Aggregation	2006	SIGMOD	0.00011057534
1,569	HELIX: Holistic Optimization for Accelerating Iterative Machine Learning	2019	VLDB	0.00010335423
1,670	MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis	2018	SIGMOD	0.00010045615
1,883	ReStore: Reusing Results of MapReduce Jobs	2012	VLDB	9.5421713e-05
2,802	General Incremental Sliding-Window Aggregation	2015	VLDB	8.1093063e-05
3,163	Multi-Query Optimization in MapReduce Framework	2014	VLDB	7.6784171e-05
4,214	The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox	2015	CIDR	6.8276099e-05
5,699	Optimizing Machine Learning Workloads in Collaborative Environments	2020	SIGMOD	6.1170243e-05
6,038	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB	5.9990929e-05
7,856	AJoin: Ad-hoc Stream Joins at Scale	2020	VLDB	5.5306709e-05
8,648	ApproxML: Efficient Approximate Ad-Hoc ML Models Through Materialization and Reuse	2019	VLDB	5.3920944e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,147	Data Management Challenges in Production Machine Learning	2017	SIGMOD
2	3,541	Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines	2022	SIGMOD
3	2,018	tf.data: A Machine Learning Data Processing Framework	2021	VLDB
4	9,245	Towards Observability for Production Machine Learning Pipelines	2022	VLDB
5	5,704	Optimizing Data Pipelines for Machine Learning in Feature Stores	2023	VLDB
6	4,240	LIMA: Fine-grained Lineage Tracing and Reuse in Machine Learning Systems	2021	SIGMOD
7	7,395	Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines	2023	SIGMOD
8	6,038	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB
9	5,699	Optimizing Machine Learning Workloads in Collaborative Environments	2020	SIGMOD
10	2,657	Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities	2021	SIGMOD