Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science

Summary: Introduces fine-grained, element-level provenance for ML preprocessing pipelines, formalizing provenance patterns for core operators. A Python library captures this lineage and supports debugging queries over real pipelines, with evaluated storage and scalability overheads. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 12759
Venue: VLDB
Year: 2021
Pagerank: 5.5000099e-05
Overall Rank: 8,050 | 44.78%
DOI: 10.14778/3436905.3436911

Incoming Non-self Citations Over Time

Authors

1. Adriane Chapman (University of Southampton)
2. Paolo Missier (Newcastle University)
3. Giulia Simonelli (Roma Tre University)
4. Riccardo Torlone (Roma Tre University)

BibTeX Citation

@article{chapman_vldb21,
        title = {{Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science}},
        author = {Chapman, Adriane and Missier, Paolo and Simonelli, Giulia and Torlone, Riccardo},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {4},
        pages = {507--520},
        doi = {10.14778/3436905.3436911},
        url = {https://doi.org/10.14778/3436905.3436911},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
8,949	Modyn: Data-Centric Machine Learning Pipeline Orchestration	2025	SIGMOD	5.3462965e-05
9,245	Towards Observability for Production Machine Learning Pipelines	2022	VLDB	5.2992628e-05
10,701	Unified Lineage System: Tracking Data Provenance at Scale	2025	SIGMOD	5.093636e-05
11,594	DPDS: Assisting Data Science with Data Provenance	2022	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
17	Provenance Semirings	2007	PODS	0.00059843817
1,004	Democratizing Data Science through Interactive Curation of ML Pipelines	2019	SIGMOD	0.00012701932
1,431	Towards Scalable Dataframe Systems	2020	VLDB	0.00010807221
1,571	DBNotes: A Post-It System for Relational Databases based on Provenance	2005	SIGMOD	0.00010330864
1,617	Titian: Data Provenance Support in Spark	2016	VLDB	0.00010209397
1,670	MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis	2018	SIGMOD	0.00010045615
1,798	SMOKE: Fine-grained Lineage at Interactive Speed	2018	VLDB	9.7361937e-05
2,431	noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts	2017	VLDB	8.5875984e-05
4,130	PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models	2020	SIGMOD	6.8839621e-05
4,592	Data Platform for Machine Learning	2019	SIGMOD	6.6139071e-05
4,877	TPC-DI: The First Industry Benchmark for Data Integration	2014	VLDB	6.468584e-05
7,245	The VADA Architecture for Cost-Effective Data Wrangling	2017	SIGMOD	5.6638209e-05
8,185	BugDoc: Algorithms to Debug Computational Processes	2020	SIGMOD	5.4707797e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	8,509	Hypothetical Reasoning via Provenance Abstraction	2019	SIGMOD
2	11,860	Ursprung: Provenance for Large-Scale Analytics Environments	2019	SIGMOD
3	1,871	Efficient Lineage Tracking For Scientific Workflows	2008	SIGMOD
4	1,912	Querying Data Provenance	2010	SIGMOD
5	11,668	On Optimizing the Trade-off between Privacy and Utility in Data Provenance	2021	SIGMOD
6	6,351	Tracing Lineage Beyond Relational Operators	2007	VLDB
7	8,950	OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance From Database Query Event Logs	2023	VLDB
8	2,657	Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities	2021	SIGMOD
9	11,594	DPDS: Assisting Data Science with Data Provenance	2022	VLDB
10	4,756	Improving Reproducibility of Data Science Pipelines through Transparent Provenance Capture	2020	VLDB