Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Summary: Unsupervised, corpus-driven validation infers data-domain patterns from data lakes to auto-validate data, reducing false positives on strings. Production datalake evaluation shows improved quality-issue detection vs prior methods; Azure Purview Auto-Tag. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hac779578f0d0782a
Venue: SIGMOD
Year: 2021
Pagerank: 5.4623434e-05
Overall Rank: 7,738 | 47.98%
DOI: 10.1145/3448016.3457250

Incoming Non-self Citations Over Time

Authors

1. Jie Song (University of Michigan)
2. Yeye He (Microsoft)

BibTeX Citation

@inproceedings{song_sigmod21,
        title = {{Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes}},
        author = {Song, Jie and He, Yeye},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457250},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457250},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 7 of 7 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,978	Table-GPT: Table Fine-tuned GPT for Diverse Table Tasks	2024	SIGMOD	9.2730152e-05
2,382	DeepJoin: Joinable Table Discovery with Pre-trained Language Models	2023	VLDB	8.5458532e-05
9,229	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD	5.2056825e-05
10,526	Guardrail: Automated Integrity Constraint Synthesis From Noisy Data	2026	SIGMOD	4.9793485e-05
10,853	Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models	2026	VLDB	4.9793485e-05
11,303	LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation	2025	VLDB	4.9793485e-05
11,598	Searching Data Lakes for Nested and Joined Data	2024	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 21 of 21 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050495102
95	Potter's Wheel: An Interactive Data Cleaning System	2001	VLDB	0.00034382643
160	CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies	2004	SIGMOD	0.00027837289
350	Discovering Denial Constraints	2013	VLDB	0.00020253521
435	Mining Database Structure; Or, How to Build a Data Quality Browser	2002	SIGMOD	0.0001832766
603	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015699254
649	Discovering Data Quality Rules	2008	VLDB	0.00015149865
883	HoloDetect: Few-Shot Learning for Error Detection	2019	SIGMOD	0.00013268059
1,153	Data Management Challenges in Production Machine Learning	2017	SIGMOD	0.00011798912
1,308	Automating Large-Scale Data Quality Verification	2018	VLDB	0.0001107886
1,805	Raha: A Configuration-Free Error Detection System	2019	SIGMOD	9.59842e-05
2,092	Sato: Contextual Semantic Type Detection in Tables	2020	VLDB	9.0626928e-05
2,125	Discovery of Genuine Functional Dependencies from Relational Data with Missing Values	2018	VLDB	9.0033119e-05
2,371	SCODED: Statistical Constraint Oriented Data Error Detection	2020	SIGMOD	8.5615698e-05
2,577	ClusterJoin: A Similarity Joins Framework using Map-Reduce	2014	VLDB	8.2738285e-05
2,748	Uni-Detect: A Unified Approach to Automated Error Detection in Tables	2019	SIGMOD	8.0610697e-05
2,960	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD	7.8075103e-05
4,723	Data Auditor: Exploring Data Quality and Semantics using Pattern Tableaux	2010	VLDB	6.4528647e-05
5,384	ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies	2019	SIGMOD	6.1524649e-05
6,488	Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code	2018	SIGMOD	5.76799e-05
6,898	Unit Testing Data with Deequ	2019	SIGMOD	5.6530554e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,706	TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines	2020	SIGMOD
2	10,916	AutoLiquid: Autonomic Data Layout Optimization for the Databricks Lakehouse	2026	VLDB
3	10,620	AutoDDG: Automated Dataset Description Generation using Large Language Models	2026	SIGMOD
4	4,235	Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search	2021	VLDB
5	7,526	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB
6	8,352	Towards Building Autonomous Data Services on Azure	2023	SIGMOD
7	4,424	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB
8	2,960	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD
9	9,229	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD
10	1,308	Automating Large-Scale Data Quality Verification	2018	VLDB