How do Categorical Duplicates Affect ML? A New Benchmark and Empirical Analyses

Summary: First systematic empirical study of categorical duplicates (e.g., "CA" vs "California") on ML classification: labeled corpus of 1,262 categorical columns and a 16-dataset benchmark across five classifiers and five encoders. Finds logistic regression and similarity encoding robust to duplicates while one-hot with high-capacity models degrade; provides benchmarks and actionable takeaways for AutoML and data-prep. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: hca6419be9adcb4d3
Venue: VLDB
Year: 2024
Pagerank: 5.5909675e-05
Overall Rank: 7,076 | 52.57%
DOI: 10.14778/3648160.3648178
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Vraj Shah (IBM)
2. Thomas Parashos (California State University, Northridge)
3. Arun Kumar (University of California San Diego)

BibTeX Citation

@article{shah_vldb24,
        title = {{How do Categorical Duplicates Affect ML? A New Benchmark and Empirical Analyses}},
        author = {Shah, Vraj and Parashos, Thomas and Kumar, Arun},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {6},
        pages = {1391--1404},
        doi = {10.14778/3648160.3648178},
        url = {https://doi.org/10.14778/3648160.3648178},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
9,842	CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines	2025	VLDB	5.1064323e-05
11,121	Fault Lines: Benchmarking the Impact of Label Data Quality on ML Robustness and Fairness	2026	VLDB	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
100	HoloClean: Holistic Data Repairs with Probabilistic Inference	2017	VLDB	0.00033951027
132	Deep Entity Matching with Pre-Trained Language Models	2021	VLDB	0.00030213215
158	Deep Learning for Entity Matching: A Design Space Exploration	2018	SIGMOD	0.00028049555
323	Can Foundation Models Wrangle Your Data?	2023	VLDB	0.00021022156
517	Magellan: Toward Building Entity Matching Management Systems	2016	VLDB	0.00016946929
989	Data Cleaning: Overview and Emerging Challenges	2016	SIGMOD	0.00012638176
1,489	Synthesizing Entity Matching Rules by Examples	2018	VLDB	0.00010480559
1,993	RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation	2021	VLDB	9.2224853e-05
2,271	ZeroER: Entity Resolution using Zero Labeled Examples	2020	SIGMOD	8.7110698e-05
2,476	A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching	2020	SIGMOD	8.3932405e-05
2,746	Transform-Data-by-Example (TDE): An Extensible Search Engine for Data Transformations	2018	VLDB	8.0458815e-05
3,166	Unicorn: A Unified Multi-tasking Model for Supporting Matching Tasks in Data Integration	2023	SIGMOD	7.5597853e-05
3,680	Generating Concise Entity Matching Rules	2017	SIGMOD	7.0846793e-05
4,080	Smurf: Self-Service String Matching Using Random Forests	2019	VLDB	6.8020845e-05
4,994	Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples	2021	SIGMOD	6.3099657e-05
5,117	Towards Benchmarking Feature Type Inference for AutoML Platforms	2021	SIGMOD	6.2551509e-05
5,655	DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python	2021	SIGMOD	6.0320824e-05
5,868	ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning	2016	SIGMOD	5.948547e-05
8,047	Foofah: A Programming-By-Example System for Synthesizing Data Transformation Programs	2017	SIGMOD	5.3846335e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	13,672	Demonstrating CatDB: LLM-based Generation of Data-centric ML Pipelines	2025	SIGMOD
2	10,628	Categorical Data Clustering via Value Order Estimated Distance Metric Learning	2026	SIGMOD
3	6,164	Efficient Construction of Approximate Ad-Hoc ML models Through Materialization and Reuse	2018	VLDB
4	6,861	Serving Deep Learning Models with Deduplication from Relational Databases	2022	VLDB
5	9,842	CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines	2025	VLDB
6	886	Framework for Evaluating Clustering Algorithms in Duplicate Detection	2009	VLDB
7	3,437	Modeling and Querying Possible Repairs in Duplicate Detection	2009	VLDB
8	308	Eliminating Fuzzy Duplicates in Data Warehouses	2002	VLDB
9	5,544	MDedup: Duplicate Detection with Matching Dependencies	2020	VLDB
10	5,117	Towards Benchmarking Feature Type Inference for AutoML Platforms	2021	SIGMOD