Materialization and Reuse Optimizations for Production Data Science Pipelines

Summary: Proposes budgeted materialization to precompute and store pipeline artifacts, reducing redundant data processing in retraining ML pipelines. Introduces a DAG-based reuse planner to fuse pipelines and reuse artifacts, delivering up to 10x training speedups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: haebe808d328e1066
Venue: SIGMOD
Year: 2022
Pagerank: 5.8446676e-05
Overall Rank: 6,222 | 58.19%
DOI: 10.1145/3514221.3526186

Incoming Non-self Citations Over Time

Authors

1. Behrouz Derakhshan (German National Research Center for Information Technology)
2. Alireza Rezaei Mahdiraji (Yara Digital Production)
3. Zoi Kaoudi (Technical University of Berlin)
4. Tilmann Rabl (Hasso Plattner Institute)
5. Volker Markl (German National Research Center for Information Technology; Technical University of Berlin)

BibTeX Citation

@inproceedings{derakhshan_sigmod22,
        title = {{Materialization and Reuse Optimizations for Production Data Science Pipelines}},
        author = {Derakhshan, Behrouz and Mahdiraji, Alireza Rezaei and Kaoudi, Zoi and Rabl, Tilmann and Markl, Volker},
        series = {{SIGMOD} '22},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3514221.3526186},
        url = {https://dl.acm.org/doi/10.1145/3514221.3526186},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,774	Optimizing Data Pipelines for Machine Learning in Feature Stores	2023	VLDB	5.9959266e-05
7,544	Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines	2023	SIGMOD	5.496984e-05
9,079	Scheduling Data Processing Pipelines for Incremental Training on MLP-based Recommendation Models	2025	SIGMOD	5.2258409e-05
10,941	IMLane: Composable Framework for Efficient AI Function Execution in Database Engine	2026	VLDB	4.9769913e-05
11,554	Pasta: A Cost-Based Optimizer for Generating Pipelining Schedules for Dataflow DAGs	2024	SIGMOD	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 18 of 18 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
83	Space-Efficient Online Computation of Quantile Summaries	2001	SIGMOD	0.00035962466
129	Efficient and Extensible Algorithms for Multi Query Optimization	2000	SIGMOD	0.00030395767
579	Incremental Knowledge Base Construction Using DeepDive	2015	VLDB	0.00016083582
654	Materialization Optimizations for Feature Selection Workloads	2014	SIGMOD	0.00015096817
823	MRShare: Sharing Across Multiple Queries in MapReduce	2010	VLDB	0.00013642066
831	Pipelining in Multi-Query Optimization	2001	PODS	0.00013583882
1,355	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00010921447
1,383	On-the-Fly Sharing for Streamed Aggregation	2006	SIGMOD	0.00010842596
1,568	HELIX: Holistic Optimization for Accelerating Iterative Machine Learning	2019	VLDB	0.00010208225
1,692	MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis	2018	SIGMOD	9.8526476e-05
1,925	ReStore: Reusing Results of MapReduce Jobs	2012	VLDB	9.3643089e-05
2,740	General Incremental Sliding-Window Aggregation	2015	VLDB	8.0682988e-05
3,217	Multi-Query Optimization in MapReduce Framework	2014	VLDB	7.5199872e-05
4,300	The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox	2015	CIDR	6.676283e-05
5,794	Optimizing Machine Learning Workloads in Collaborative Environments	2020	SIGMOD	5.9906542e-05
6,146	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB	5.869373e-05
8,019	AJoin: Ad-hoc Stream Joins at Scale	2020	VLDB	5.4045347e-05
8,815	ApproxML: Efficient Approximate Ad-Hoc ML Models Through Materialization and Reuse	2019	VLDB	5.2703166e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,153	Data Management Challenges in Production Machine Learning	2017	SIGMOD
2	3,601	Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines	2022	SIGMOD
3	2,055	tf.data: A Machine Learning Data Processing Framework	2021	VLDB
4	9,426	Towards Observability for Production Machine Learning Pipelines	2022	VLDB
5	5,774	Optimizing Data Pipelines for Machine Learning in Feature Stores	2023	VLDB
6	4,334	LIMA: Fine-grained Lineage Tracing and Reuse in Machine Learning Systems	2021	SIGMOD
7	7,544	Automating and Optimizing Data-Centric What-If Analyses on Native Machine Learning Pipelines	2023	SIGMOD
8	6,146	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB
9	5,794	Optimizing Machine Learning Workloads in Collaborative Environments	2020	SIGMOD
10	2,189	Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities	2021	SIGMOD