Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Summary: Unsupervised, corpus-driven validation infers data-domain patterns from data lakes to auto-validate data, reducing false positives on strings. Production datalake evaluation shows improved quality-issue detection vs prior methods; Azure Purview Auto-Tag. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hac779578f0d0782a
Venue: SIGMOD
Year: 2021
Pagerank: 5.4424293e-05
Overall Rank: 7,755 | 48.02%
DOI: 10.1145/3448016.3457250

Incoming Non-self Citations Over Time

Authors

1. Jie Song (University of Michigan)
2. Yeye He (Microsoft)

BibTeX Citation

@inproceedings{song_sigmod21,
        title = {{Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes}},
        author = {Song, Jie and He, Yeye},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457250},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457250},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 7 of 7 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,669	Table-GPT: Table Fine-tuned GPT for Diverse Table Tasks	2024	SIGMOD	9.9157758e-05
2,370	DeepJoin: Joinable Table Discovery with Pre-trained Language Models	2023	VLDB	8.5469172e-05
9,263	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD	5.1860131e-05
10,603	Guardrail: Automated Integrity Constraint Synthesis From Noisy Data	2026	SIGMOD	4.9605343e-05
10,923	Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models	2026	VLDB	4.9605343e-05
11,365	LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation	2025	VLDB	4.9605343e-05
11,651	Searching Data Lakes for Nested and Joined Data	2024	VLDB	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 21 of 21 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050364918
95	Potter's Wheel: An Interactive Data Cleaning System	2001	VLDB	0.00034319763
160	CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies	2004	SIGMOD	0.00027775994
348	Discovering Denial Constraints	2013	VLDB	0.00020216677
435	Mining Database Structure; Or, How to Build a Data Quality Browser	2002	SIGMOD	0.00018277436
602	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015670535
653	Discovering Data Quality Rules	2008	VLDB	0.00015102328
853	HoloDetect: Few-Shot Learning for Error Detection	2019	SIGMOD	0.00013437314
1,151	Data Management Challenges in Production Machine Learning	2017	SIGMOD	0.000117662
1,278	Automating Large-Scale Data Quality Verification	2018	VLDB	0.00011208469
1,731	Raha: A Configuration-Free Error Detection System	2019	SIGMOD	9.7585913e-05
1,985	Sato: Contextual Semantic Type Detection in Tables	2020	VLDB	9.2378767e-05
2,131	Discovery of Genuine Functional Dependencies from Relational Data with Missing Values	2018	VLDB	8.9867472e-05
2,373	SCODED: Statistical Constraint Oriented Data Error Detection	2020	SIGMOD	8.5434538e-05
2,581	ClusterJoin: A Similarity Joins Framework using Map-Reduce	2014	VLDB	8.2491224e-05
2,752	Uni-Detect: A Unified Approach to Automated Error Detection in Tables	2019	SIGMOD	8.0387193e-05
2,960	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD	7.7967245e-05
4,741	Data Auditor: Exploring Data Quality and Semantics using Pattern Tableaux	2010	VLDB	6.4290861e-05
5,371	ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies	2019	SIGMOD	6.1428928e-05
6,467	Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code	2018	SIGMOD	5.7580747e-05
6,911	Unit Testing Data with Deequ	2019	SIGMOD	5.6335281e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,711	TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines	2020	SIGMOD
2	10,986	AutoLiquid: Autonomic Data Layout Optimization for the Databricks Lakehouse	2026	VLDB
3	10,692	AutoDDG: Automated Dataset Description Generation using Large Language Models	2026	SIGMOD
4	4,005	Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search	2021	VLDB
5	7,548	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB
6	8,372	Towards Building Autonomous Data Services on Azure	2023	SIGMOD
7	2,960	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD
8	4,444	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB
9	9,263	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD
10	1,278	Automating Large-Scale Data Quality Verification	2018	VLDB