R2D2: Reducing Redundancy and Duplication in Data Lakes

Summary: R2D2 tackles table-level containment in data lakes with a three-stage pipeline: schema containment graph, min-max pruning, and content-level pruning—for scalable detection. It trims storage and access costs by deleting redundant datasets and reconstructing on demand under latency bounds; built on Spark (Azure Databricks/ADLS Gen2, AWS) for TB-scale lakes. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6769
Venue: SIGMOD
Year: 2023
Pagerank: 4.427232e-05
Overall Rank: 8,910 | 38.02%
DOI: 10.1145/3626762

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
10,867	T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data	2025	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
221	Deep Entity Matching with Pre-Trained Language Models	2021	VLDB	0.00033121824
1,178	Table Union Search on Open Data	2018	VLDB	0.00013468118
1,187	JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes	2019	SIGMOD	0.00013443639
1,565	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011345567
1,644	Finding Related Tables in Data Lakes for Interactive Data Science	2020	SIGMOD	0.00011041787
2,141	LSH Ensemble: Internet-Scale Domain Search	2016	VLDB	9.4542625e-05
2,836	Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning	2023	VLDB	8.0443826e-05
3,000	SANTOS: Relationship-based Semantic Table Union Search	2023	SIGMOD	7.7462128e-05
4,353	Overlap Set Similarity Joins with Theoretical Guarantees	2018	SIGMOD	6.263585e-05
5,058	A Demo of the Data Civilizer System	2017	SIGMOD	5.7280139e-05
5,179	SilkMoth: An Efficient Method for Finding Related Sets with Maximum Matching Constraints	2017	VLDB	5.6428428e-05
5,271	ORPHEUSDB: A Lightweight Approach to Relational Dataset Versioning	2017	SIGMOD	5.5941385e-05
5,794	Discovering Related Data At Scale	2021	VLDB	5.3245122e-05
6,438	RONIN: Data Lake Exploration	2021	VLDB	5.0620163e-05
7,635	Allign: Aligning All-Pair Near-Duplicate Passages in Long Texts	2021	SIGMOD	4.6908858e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
5,236	Online Deduplication for Databases	2017	SIGMOD	5.611324e-05
280	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB	0.00029113044
7,059	Adaptive and Robust Query Execution for Lakehouses at Scale	2024	VLDB	4.8477825e-05
2,386	Leveraging Aggregate Constraints For Deduplication	2007	SIGMOD	8.9231895e-05
9,232	AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes	2025	SIGMOD	4.3690661e-05
1,482	Automating Large-Scale Data Quality Verification	2018	VLDB	0.00011725533
9,479	Data Imputation with Limited Data Redundancy Using Data Lakes	2025	VLDB	4.3341665e-05
3,528	Distributed Data Deduplication	2016	VLDB	7.0066139e-05
7,907	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB	4.6205839e-05
7,061	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB	4.8463881e-05