R2D2: Reducing Redundancy and Duplication in Data Lakes

Summary: R2D2 tackles table-level containment in data lakes with a three-stage pipeline: schema containment graph, min-max pruning, and content-level pruning—for scalable detection. It trims storage and access costs by deleting redundant datasets and reconstructing on demand under latency bounds; built on Spark (Azure Databricks/ADLS Gen2, AWS) for TB-scale lakes. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6770
Venue: SIGMOD
Year: 2023
Pagerank: 4.4229886e-05
Overall Rank: 8,912 | 38.07%
DOI: 10.1145/3626762

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
10,871	T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data	2025	VLDB	4.1905499e-05

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
219	Deep Entity Matching with Pre-Trained Language Models	2021	VLDB	0.00033354456
1,179	Table Union Search on Open Data	2018	VLDB	0.00013458551
1,185	JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes	2019	SIGMOD	0.00013432692
1,565	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011336478
1,643	Finding Related Tables in Data Lakes for Interactive Data Science	2020	SIGMOD	0.00011031534
2,142	LSH Ensemble: Internet-Scale Domain Search	2016	VLDB	9.4461701e-05
2,842	Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning	2023	VLDB	8.0366354e-05
3,001	SANTOS: Relationship-based Semantic Table Union Search	2023	SIGMOD	7.739698e-05
4,350	Overlap Set Similarity Joins with Theoretical Guarantees	2018	SIGMOD	6.2576191e-05
5,056	A Demo of the Data Civilizer System	2017	SIGMOD	5.7225065e-05
5,180	SilkMoth: An Efficient Method for Finding Related Sets with Maximum Matching Constraints	2017	VLDB	5.6377866e-05
5,276	ORPHEUSDB: A Lightweight Approach to Relational Dataset Versioning	2017	SIGMOD	5.5887569e-05
5,805	Discovering Related Data At Scale	2021	VLDB	5.3197639e-05
6,432	RONIN: Data Lake Exploration	2021	VLDB	5.0571585e-05
7,636	Allign: Aligning All-Pair Near-Duplicate Passages in Long Texts	2021	SIGMOD	4.6863871e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
5,240	Online Deduplication for Databases	2017	SIGMOD	5.6060424e-05
279	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB	0.00029141798
6,683	Adaptive and Robust Query Execution for Lakehouses at Scale	2024	VLDB	4.9593505e-05
2,385	Leveraging Aggregate Constraints For Deduplication	2007	SIGMOD	8.9167648e-05
9,236	AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes	2025	SIGMOD	4.3648789e-05
1,481	Automating Large-Scale Data Quality Verification	2018	VLDB	0.00011715754
9,481	Data Imputation with Limited Data Redundancy Using Data Lakes	2025	VLDB	4.3300131e-05
3,526	Distributed Data Deduplication	2016	VLDB	7.0056559e-05
7,907	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB	4.6161532e-05
7,055	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB	4.8428976e-05