Distributed Data Deduplication

Summary: Distributed data deduplication in a shared-nothing setting; leverages parallelism to prune pairwise comparisons after blocking. Dis-Dedup—a distribution strategy that minimizes the maximum per-node workload with theoretical guarantees; experiments on synthetic and real data show scalable speedups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11562
Venue: VLDB
Year: 2016
Pagerank: 7.983961e-05
Overall Rank: 2,893 | 80.16%
DOI: 10.14778/2983200.2983206

Incoming Non-self Citations Over Time

Authors

1. Xu Chu (University of Waterloo)
2. Ihab F. Ilyas (University of Waterloo)
3. Paraschos Koutris (University of Wisconsin)

BibTeX Citation

@article{chu_vldb16,
        title = {{Distributed Data Deduplication}},
        author = {Chu, Xu and Ilyas, Ihab F. and Koutris, Paraschos},
        journal = {PVLDB},
        series = {{VLDB} '16},
        volume = {9},
        number = {11},
        pages = {864--875},
        doi = {10.14778/2983200.2983206},
        url = {https://doi.org/10.14778/2983200.2983206},
        year = {2016}
}

Incoming Citations (Sorted by Pagerank)

Showing 16 of 16 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
529	Magellan: Toward Building Entity Matching Management Systems	2016	VLDB	0.00017096361
1,643	Falcon: Scaling Up Hands-Off Crowdsourced Entity Matching to Build Cloud Services	2017	SIGMOD	0.00010134956
5,052	Distributed implementations of dependency discovery algorithms	2019	VLDB	6.3846183e-05
5,134	Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples	2021	SIGMOD	6.3532024e-05
6,392	Qualitative Data Cleaning	2016	VLDB	5.8883971e-05
6,403	Parallel Discrepancy Detection and Incremental Detection	2021	VLDB	5.8859374e-05
6,708	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB	5.7964983e-05
7,111	Submodularity of Distributed Join Computation	2018	SIGMOD	5.69924e-05
7,232	Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications	2023	SIGMOD	5.6659017e-05
8,225	Deducing Certain Fixes to Graphs	2019	VLDB	5.4624971e-05
8,318	Optimizing Declarative Graph Queries at Large Scale	2019	SIGMOD	5.4548766e-05
8,908	Privacy and Accuracy-Aware AI/ML Model Deduplication	2025	SIGMOD	5.3483178e-05
9,627	Making It Tractable to Catch Duplicates and Conflicts in Graphs	2023	SIGMOD	5.2434488e-05
9,997	HyperBlocker: Accelerating Rule-based Blocking in Entity Resolution using GPUs	2025	VLDB	5.1814573e-05
11,636	New Algorithms for Monotone Classification	2021	PODS	5.093636e-05
11,898	Entity Matching with Active Monotone Classification	2018	PODS	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
58	The Merge/Purge Problem for Large Databases	1995	SIGMOD	0.00040116748
248	Evaluation of entity resolution approaches on real-world match problems	2010	VLDB	0.00023278354
306	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB	0.00021839661
356	Efficient Parallel Set-Similarity Joins Using MapReduce	2010	SIGMOD	0.00020303289
481	Practical Skew Handling in Parallel Joins	1992	VLDB	0.00017780716
626	Entity Resolution: Theory, Practice & Open Challenges	2012	VLDB	0.00015656958
843	Processing Theta-Joins using MapReduce*	2011	SIGMOD	0.00013666161
1,320	From Theory to Practice: Efficient Join Query Evaluation in a Parallel Database System	2015	SIGMOD	0.00011166426
1,448	Skew in Parallel Query Processing	2014	PODS	0.00010758872
2,055	Dedoop: Efficient Deduplication with Hadoop	2012	VLDB	9.2512023e-05
2,567	ClusterJoin: A Similarity Joins Framework using Map-Reduce	2014	VLDB	8.4098241e-05
2,804	Demonstration of the Myria Big Data Management Service	2014	SIGMOD	8.1075524e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,718	Sharing Aggregate Computation for Distributed Queries	2007	SIGMOD
2	3,371	Modeling and Querying Possible Repairs in Duplicate Detection	2009	VLDB
3	7,585	Query Centric Partitioning and Allocation for Partially Replicated Database Systems	2017	SIGMOD
4	5,052	Distributed implementations of dependency discovery algorithms	2019	VLDB
5	6,403	Parallel Discrepancy Detection and Incremental Detection	2021	VLDB
6	3,983	Crowd-Based Deduplication: An Adaptive Approach	2015	SIGMOD
7	1,637	On the Design and Scalability of Distributed Shared-Data Databases	2015	SIGMOD
8	3,200	Locality-aware Partitioning in Parallel Database Systems	2015	SIGMOD
9	3,669	Distributing A Database For Parallelism	1983	SIGMOD
10	5,021	Online Deduplication for Databases	2017	SIGMOD