OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance From Database Query Event Logs

Summary: OneProvenance: a log-based system that efficiently extracts coarse-grained provenance from DB query event logs by recovering query execution dependencies via lightweight log analysis and novel event transformations. Adds filtering optimizations to cut noise and capture execution dependencies, yielding up to ~18× faster extraction than prior work and deployed at scale in Microsoft Purview. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13383
Venue: VLDB
Year: 2023
Pagerank: 5.345946e-05
Overall Rank: 8,950 | 38.60%
DOI: 10.14778/3611550.3611555

Incoming Non-self Citations Over Time

Authors

1. Fotis Psallidas (Microsoft)
2. Ashvin Agrawal (Microsoft)
3. Chandru Sugunan (Snowflake)
4. Khaled Ibrahim (Microsoft)
5. Konstantinos Karanasos (Meta)
6. Jesús Camacho-Rodríguez (Microsoft)
7. Avrilia Floratou (Microsoft)
8. Carlo Curino (Microsoft)
9. Raghu Ramakrishnan (Microsoft)

BibTeX Citation

@article{psallidas_vldb23,
        title = {{OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance From Database Query Event Logs}},
        author = {Psallidas, Fotis and Agrawal, Ashvin and Sugunan, Chandru and Ibrahim, Khaled and Karanasos, Konstantinos and Camacho-Rodríguez, Jesús and Floratou, Avrilia and Curino, Carlo and Ramakrishnan, Raghu},
        journal = {PVLDB},
        series = {{VLDB} '23},
        volume = {16},
        number = {12},
        pages = {3662--3675},
        doi = {10.14778/3611550.3611555},
        url = {https://doi.org/10.14778/3611550.3611555},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,398	Real-time Workload Pattern Analysis for Large-scale Cloud Databases	2023	VLDB	6.7248611e-05
10,701	Unified Lineage System: Tracking Data Provenance at Scale	2025	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 23 of 23 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
125	Trio: A System for Integrated Management of Data, Accuracy, and Lineage	2005	CIDR	0.00030932014
493	Data Integration for the Relational Web	2009	VLDB	0.00017558709
514	Goods: Organizing Google's Datasets	2016	SIGMOD	0.00017178673
531	An Annotation Management System for Relational Databases	2004	VLDB	0.00017083233
714	Guided Data Repair	2011	VLDB	0.00014662041
749	Data Lake Management: Challenges and Opportunities	2019	VLDB	0.00014379989
878	Table Union Search on Open Data	2018	VLDB	0.00013463374
963	The Data Civilizer System	2017	CIDR	0.00012935145
1,077	DataHub: Collaborative Data Science & Dataset Version Management at Scale	2015	CIDR	0.00012269438
1,349	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011069122
1,475	VisTrails: Visualization meets Data Management	2006	SIGMOD	0.00010662942
1,495	LSH Ensemble: Internet-Scale Domain Search	2016	VLDB	0.00010571481
1,599	Update Exchange with Mappings and Provenance	2007	VLDB	0.00010241307
1,649	Bootstrapping Pay-As-You-Go Data Integration Systems	2008	SIGMOD	0.00010117306
1,786	Putting Lipstick on Pig: Enabling Database-style Workflow Provenance	2012	VLDB	9.7690421e-05
1,798	SMOKE: Fine-grained Lineage at Interactive Speed	2018	VLDB	9.7361937e-05
1,892	Procedural Extensions of SQL: Understanding their usage in the wild	2021	VLDB	9.5277793e-05
1,902	Ground: A Data Context Service	2017	CIDR	9.506714e-05
2,038	Graceful Database Schema Evolution: the PRISM Workbench	2008	VLDB	9.2727673e-05
4,201	CLAMS: Bringing Quality to Data Lakes	2016	SIGMOD	6.8355878e-05
4,756	Improving Reproducibility of Data Science Pipelines through Transparent Provenance Capture	2020	VLDB	6.5221401e-05
5,509	Ibis: A Provenance Manager for Multi-Layer Systems	2011	CIDR	6.192039e-05
7,878	Dependency-Driven Analytics: a Compass for Uncharted Data Oceans	2017	CIDR	5.5246167e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,602	Provenance for Natural Language Queries	2017	VLDB
2	8,050	Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science	2021	VLDB
3	806	Provenance for Aggregate Queries	2011	PODS
4	8,976	Computing How-Provenance for SPARQL Queries via Query Rewriting	2021	VLDB
5	2,807	Data Provenance at Internet Scale: Architecture, Experiences, and the Road Ahead	2017	CIDR
6	8,145	You Say ‘What’, I Hear ‘Where’ and ‘Why’ — (Mis-)Interpreting SQL to Derive Fine-Grained Provenance	2018	VLDB
7	7,977	Interactive Query Explanations Using Fine Grained Provenance	2022	SIGMOD
8	1,912	Querying Data Provenance	2010	SIGMOD
9	6,358	On Provenance Minimization	2011	PODS
10	11,860	Ursprung: Provenance for Large-Scale Analytics Environments	2019	SIGMOD