Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Summary: Unsupervised, corpus-driven validation infers data-domain patterns from data lakes to auto-validate data, reducing false positives on strings. Production datalake evaluation shows improved quality-issue detection vs prior methods; Azure Purview Auto-Tag. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6203
Venue: SIGMOD
Year: 2021
Pagerank: 5.5669307e-05
Overall Rank: 7,690 | 47.25%
DOI: 10.1145/3448016.3457250

Incoming Non-self Citations Over Time

Authors

1. Jie Song (University of Michigan)
2. Yeye He (Microsoft)

BibTeX Citation

@inproceedings{song_sigmod21,
        title = {{Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes}},
        author = {Song, Jie and He, Yeye},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457250},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457250},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 6 of 6 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,099	Table-GPT: Table Fine-tuned GPT for Diverse Table Tasks	2024	SIGMOD	9.1682353e-05
3,073	DeepJoin: Joinable Table Discovery with Pre-trained Language Models	2023	VLDB	7.785842e-05
10,317	Guardrail: Automated Integrity Constraint Synthesis From Noisy Data	2026	SIGMOD	5.093636e-05
10,785	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD	5.093636e-05
10,909	LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation	2025	VLDB	5.093636e-05
11,270	Searching Data Lakes for Nested and Joined Data	2024	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 21 of 21 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00051174276
94	Potter's Wheel: An Interactive Data Cleaning System	2001	VLDB	0.00034616103
159	CORDS: Automatic Discovery of Correlations and Soft Functional Dependencies	2004	SIGMOD	0.00028129426
376	Discovering Denial Constraints	2013	VLDB	0.00019677674
432	Mining Database Structure; Or, How to Build a Data Quality Browser	2002	SIGMOD	0.00018572055
618	A Hybrid Approach to Functional Dependency Discovery	2016	SIGMOD	0.00015711835
647	Discovering Data Quality Rules	2008	VLDB	0.00015334666
946	HoloDetect: Few-Shot Learning for Error Detection	2019	SIGMOD	0.00013054126
1,147	Data Management Challenges in Production Machine Learning	2017	SIGMOD	0.00011974846
1,350	Automating Large-Scale Data Quality Verification	2018	VLDB	0.00011065626
2,123	Discovery of Genuine Functional Dependencies from Relational Data with Missing Values	2018	VLDB	9.1372798e-05
2,223	Sato: Contextual Semantic Type Detection in Tables	2020	VLDB	8.9189986e-05
2,297	Raha: A Configuration-Free Error Detection System	2019	SIGMOD	8.7897221e-05
2,567	ClusterJoin: A Similarity Joins Framework using Map-Reduce	2014	VLDB	8.4098241e-05
2,728	Uni-Detect: A Unified Approach to Automated Error Detection in Tables	2019	SIGMOD	8.1995954e-05
3,004	SCODED: Statistical Constraint Oriented Data Error Detection	2020	SIGMOD	7.8608629e-05
3,147	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD	7.7077175e-05
4,642	Data Auditor: Exploring Data Quality and Semantics using Pattern Tableaux	2010	VLDB	6.5908277e-05
5,268	ANMAT: Automatic Knowledge Discovery and Error Detection through Pattern Functional Dependencies	2019	SIGMOD	6.2923403e-05
6,436	Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code	2018	SIGMOD	5.8793793e-05
6,801	Unit Testing Data with Deequ	2019	SIGMOD	5.7690726e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,629	Auto-BI: Automatically Build BI-Models Leveraging Local Join Prediction and Global Schema Graph	2023	VLDB
2	3,630	TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines	2020	SIGMOD
3	10,431	AutoDDG: Automated Dataset Description Generation using Large Language Models	2026	SIGMOD
4	4,875	Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search	2021	VLDB
5	9,380	Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence	2025	VLDB
6	8,193	Towards Building Autonomous Data Services on Azure	2023	SIGMOD
7	4,412	Auto-Transform: Learning-to-Transform by Patterns	2020	VLDB
8	3,147	Auto-Detect: Data-Driven Error Detection in Tables	2018	SIGMOD
9	10,785	Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables	2025	SIGMOD
10	1,350	Automating Large-Scale Data Quality Verification	2018	VLDB