How do Categorical Duplicates Affect ML? A New Benchmark and Empirical Analyses

Summary: First systematic empirical study of categorical duplicates (e.g., "CA" vs "California") on ML classification: labeled corpus of 1,262 categorical columns and a 16-dataset benchmark across five classifiers and five encoders. Finds logistic regression and similarity encoding robust to duplicates while one-hot with high-capacity models degrade; provides benchmarks and actionable takeaways for AutoML and data-prep. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: hca6419be9adcb4d3
Venue: VLDB
Year: 2024
Pagerank: 5.6083188e-05
Overall Rank: 7,068 | 52.48%
DOI: 10.14778/3648160.3648178

Incoming Non-self Citations Over Time

Authors

1. Vraj Shah (IBM)
2. Thomas Parashos (California State University, Northridge)
3. Arun Kumar (University of California San Diego)

BibTeX Citation

@article{shah_vldb24,
        title = {{How do Categorical Duplicates Affect ML? A New Benchmark and Empirical Analyses}},
        author = {Shah, Vraj and Parashos, Thomas and Kumar, Arun},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {6},
        pages = {1391--1404},
        doi = {10.14778/3648160.3648178},
        url = {https://doi.org/10.14778/3648160.3648178},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
11,051	Fault Lines: Benchmarking the Impact of Label Data Quality on ML Robustness and Fairness	2026	VLDB	4.9793485e-05
11,284	CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines	2025	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
104	HoloClean: Holistic Data Repairs with Probabilistic Inference	2017	VLDB	0.00033690989
134	Deep Entity Matching with Pre-Trained Language Models	2021	VLDB	0.00030043481
158	Deep Learning for Entity Matching: A Design Space Exploration	2018	SIGMOD	0.00028046388
329	Can Foundation Models Wrangle Your Data?	2023	VLDB	0.00020858443
530	Magellan: Toward Building Entity Matching Management Systems	2016	VLDB	0.00016855162
1,043	Data Cleaning: Overview and Emerging Challenges	2016	SIGMOD	0.00012335114
1,493	Synthesizing Entity Matching Rules by Examples	2018	VLDB	0.00010505174
1,991	RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation	2021	VLDB	9.2383849e-05
2,323	ZeroER: Entity Resolution using Zero Labeled Examples	2020	SIGMOD	8.6348884e-05
2,475	A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching	2020	SIGMOD	8.410678e-05
2,745	Transform-Data-by-Example (TDE): An Extensible Search Engine for Data Transformations	2018	VLDB	8.0668395e-05
3,473	Unicorn: A Unified Multi-tasking Model for Supporting Matching Tasks in Data Integration	2023	SIGMOD	7.2728706e-05
3,681	Generating Concise Entity Matching Rules	2017	SIGMOD	7.1014441e-05
4,107	Smurf: Self-Service String Matching Using Random Forests	2019	VLDB	6.8026037e-05
4,970	Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples	2021	SIGMOD	6.3329595e-05
5,100	Towards Benchmarking Feature Type Inference for AutoML Platforms	2021	SIGMOD	6.27349e-05
5,640	DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python	2021	SIGMOD	6.0548401e-05
5,881	ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning	2016	SIGMOD	5.9588636e-05
8,025	Foofah: A Programming-By-Example System for Synthesizing Data Transformation Programs	2017	SIGMOD	5.4047151e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	13,625	Demonstrating CatDB: LLM-based Generation of Data-centric ML Pipelines	2025	SIGMOD
2	10,553	Categorical Data Clustering via Value Order Estimated Distance Metric Learning	2026	SIGMOD
3	6,143	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB
4	6,844	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB
5	11,284	CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines	2025	VLDB
6	885	Framework for Evaluating Clustering Algorithms in Duplicate Detection	2009	VLDB
7	3,426	Modeling and Querying Possible Repairs in Duplicate Detection	2009	VLDB
8	307	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB
9	5,522	MDedup: Duplicate Detection with Matching Dependencies	2020	VLDB
10	5,100	Towards Benchmarking Feature Type Inference for AutoML Platforms	2021	SIGMOD