R2D2: Reducing Redundancy and Duplication in Data Lakes

Summary: R2D2 tackles table-level containment in data lakes with a three-stage pipeline: schema containment graph, min-max pruning, and content-level pruning—for scalable detection. It trims storage and access costs by deleting redundant datasets and reconstructing on demand under latency bounds; built on Spark (Azure Databricks/ADLS Gen2, AWS) for TB-scale lakes. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6831
Venue: SIGMOD
Year: 2023
Pagerank: 5.3251649e-05
Overall Rank: 9,065 | 37.81%
DOI: 10.1145/3626762

Incoming Non-self Citations Over Time

Authors

1. Raunak Shah (University of Illinois Urbana-Champaign)
2. Koyel Mukherjee (Adobe)
3. Atharv Tyagi (Adobe)
4. Sai Keerthana Karnam (Adobe)
5. Dhruv Joshi (Indian Institute of Technology Kharagpur)
6. Shivam Bhosale (Adobe)
7. Subrata Mitra (Adobe)

BibTeX Citation

@inproceedings{shah_sigmod23,
        title = {{R2D2: Reducing Redundancy and Duplication in Data Lakes}},
        author = {Shah, Raunak and Mukherjee, Koyel and Tyagi, Atharv and Karnam, Sai Keerthana and Joshi, Dhruv and Bhosale, Shivam and Mitra, Subrata},
        series = {{SIGMOD} '23},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3626762},
        url = {https://dl.acm.org/doi/10.1145/3626762},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
11,090	T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data	2025	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
141	Deep Entity Matching with Pre-Trained Language Models	2021	VLDB	0.0002964847
779	JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes	2019	SIGMOD	0.00014092047
878	Table Union Search on Open Data	2018	VLDB	0.00013463374
1,303	Finding Related Tables in Data Lakes for Interactive Data Science	2020	SIGMOD	0.0001123653
1,349	Principles of Dataset Versioning: Exploring the Recreation/Storage Tradeoff	2015	VLDB	0.00011069122
1,495	LSH Ensemble: Internet-Scale Domain Search	2016	VLDB	0.00010571481
2,192	Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning	2023	VLDB	8.9767241e-05
2,315	SANTOS: Relationship-based Semantic Table Union Search	2023	SIGMOD	8.7605101e-05
3,724	Overlap Set Similarity Joins with Theoretical Guarantees	2018	SIGMOD	7.1715735e-05
4,707	SILKMOTH: An Efficient Method for Finding Related Sets with Maximum Matching Constraints	2017	VLDB	6.552423e-05
5,026	ORPHEUSDB: A Lightweight Approach to Relational Dataset Versioning	2017	SIGMOD	6.3970621e-05
5,344	Discovering Related Data At Scale	2021	VLDB	6.2591104e-05
5,644	A Demo of the Data Civilizer System	2017	SIGMOD	6.1378898e-05
5,956	RONIN: Data Lake Exploration	2021	VLDB	6.0287066e-05
7,676	Allign: Aligning All-Pair Near-Duplicate Passages in Long Texts	2021	SIGMOD	5.5686107e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,021	Online Deduplication for Databases	2017	SIGMOD
2	306	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB
3	5,983	Adaptive and Robust Query Execution for Lakehouses at Scale	2024	VLDB
4	2,410	Leveraging Aggregate Constraints For Deduplication	2007	SIGMOD
5	9,378	AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes	2025	SIGMOD
6	1,350	Automating Large-Scale Data Quality Verification	2018	VLDB
7	9,550	Data Imputation with Limited Data Redundancy Using Data Lakes	2025	VLDB
8	2,893	Distributed Data Deduplication	2016	VLDB
9	7,780	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB
10	6,708	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB