Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Summary: Unsupervised, corpus-driven validation infers data-domain patterns from data lakes to auto-validate data, reducing false positives on strings. Production datalake evaluation shows improved quality-issue detection vs prior methods; Azure Purview Auto-Tag. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hac779578f0d0782a
Venue: SIGMOD
Year: 2021
Pagerank: 5.4599258e-05
Overall Rank: 7,742 | 47.97%
DOI: 10.1145/3448016.3457250

Incoming Non-self Citations Over Time

Authors

1. Jie Song (University of Michigan)
2. Yeye He (Microsoft)

BibTeX Citation

@inproceedings{song_sigmod21,
        title = {{Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes}},
        author = {Song, Jie and He, Yeye},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457250},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457250},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 7 of 7 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,977	Table-GPT: Table Fine-tuned GPT for Diverse Table Tasks	2024	SIGMOD	9.2760522e-05
2,378	DeepJoin: Joinable Table Discovery with Pre-trained Language Models	2023	VLDB	8.5495442e-05
9,239	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD	5.2032182e-05
10,537	Guardrail: Automated Integrity Constraint Synthesis From Noisy Data	2026	SIGMOD	4.9769913e-05
10,862	Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models	2026	VLDB	4.9769913e-05
11,311	LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation	2025	VLDB	4.9769913e-05
11,604	Searching Data Lakes for Nested and Joined Data	2024	VLDB	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 21 of 21 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050475202
95	Potter's Wheel: An Interactive Data Cleaning System	2001	VLDB	0.00034367518
160	CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies	2004	SIGMOD	0.00027827605
350	Discovering Denial Constraints	2013	VLDB	0.00020244085
435	Mining Database Structure; Or, How to Build a Data Quality Browser	2002	SIGMOD	0.0001831946
603	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015692059
650	Discovering Data Quality Rules	2008	VLDB	0.00015142855
884	HoloDetect: Few-Shot Learning for Error Detection	2019	SIGMOD	0.00013263269
1,153	Data Management Challenges in Production Machine Learning	2017	SIGMOD	0.00011793347
1,308	Automating Large-Scale Data Quality Verification	2018	VLDB	0.00011073863
1,805	Raha: A Configuration-Free Error Detection System	2019	SIGMOD	9.5938877e-05
2,093	Sato: Contextual Semantic Type Detection in Tables	2020	VLDB	9.0593586e-05
2,127	Discovery of Genuine Functional Dependencies from Relational Data with Missing Values	2018	VLDB	8.9990594e-05
2,372	SCODED: Statistical Constraint Oriented Data Error Detection	2020	SIGMOD	8.5575299e-05
2,577	ClusterJoin: A Similarity Joins Framework using Map-Reduce	2014	VLDB	8.2699584e-05
2,748	Uni-Detect: A Unified Approach to Automated Error Detection in Tables	2019	SIGMOD	8.0572603e-05
2,962	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD	7.8038782e-05
4,724	Data Auditor: Exploring Data Quality and Semantics using Pattern Tableaux	2010	VLDB	6.4498285e-05
5,391	ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies	2019	SIGMOD	6.1495611e-05
6,490	Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code	2018	SIGMOD	5.7653145e-05
6,900	Unit Testing Data with Deequ	2019	SIGMOD	5.6503837e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,706	TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines	2020	SIGMOD
2	10,925	AutoLiquid: Autonomic Data Layout Optimization for the Databricks Lakehouse	2026	VLDB
3	10,631	AutoDDG: Automated Dataset Description Generation using Large Language Models	2026	SIGMOD
4	4,235	Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search	2021	VLDB
5	7,531	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB
6	8,357	Towards Building Autonomous Data Services on Azure	2023	SIGMOD
7	2,962	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD
8	4,426	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB
9	9,239	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD
10	1,308	Automating Large-Scale Data Quality Verification	2018	VLDB