DBScholar

Back to papers

Automating Large-Scale Data Quality Verification

Summary: Automates quality verification with a declarative API that blends standard constraints and user code to enable data unit tests. Translates validation to Spark aggregations, enables checks, and uses ML for constraint suggestions and anomaly detection. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
hb2441161cf78104e
Venue
VLDB
Year
2018
Pagerank
0.00011073863
Overall Rank
1,308 | 91.22%
DOI
10.14778/3229863.3229867
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{schelter_vldb18,
        title = {{Automating Large-Scale Data Quality Verification}},
        author = {Schelter, Sebastian and Lange, Dustin and Schmidt, Philipp and Celikel, Meltem and Biessmann, Felix and Grafberger, Andreas},
        journal = {PVLDB},
        series = {{VLDB} '18},
        volume = {11},
        number = {12},
        pages = {1781--1794},
        doi = {10.14778/3229863.3229867},
        url = {https://doi.org/10.14778/3229863.3229867},
        year = {2018}
}

Incoming Citations (Sorted by Pagerank)

Showing 28 of 28 citing papers.

Rank Citing Paper Year Venue Pagerank
1,669 SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle 2020 CIDR 9.9324573e-05
1,778 Annotating Columns with Pre-trained Language Models 2022 SIGMOD 9.6578031e-05
2,329 SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging 2021 SIGMOD 8.6268411e-05
2,642 Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks 2020 SIGMOD 8.1751637e-05
2,748 Uni-Detect: A Unified Approach to Automated Error Detection in Tables 2019 SIGMOD 8.0572603e-05
3,706 TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines 2020 SIGMOD 7.0796066e-05
4,678 spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines 2024 VLDB 6.4720352e-05
5,102 Towards Benchmarking Feature Type Inference for AutoML Platforms 2021 SIGMOD 6.2705617e-05
5,393 SchemaPile: A Large Collection of Relational Database Schemas 2024 SIGMOD 6.1494131e-05
5,574 Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications 2023 SIGMOD 6.0773771e-05
6,327 Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines 2021 CIDR 5.8111081e-05
6,666 UPLIFT: Parallelization Strategies for Feature Transformations in Machine Learning Workloads 2022 VLDB 5.7144587e-05
6,900 Unit Testing Data with Deequ 2019 SIGMOD 5.6503837e-05
7,742 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD 5.4599258e-05
8,403 Deducing Certain Fixes to Graphs 2019 VLDB 5.3374114e-05
9,027 Complaint-Driven Training Data Debugging at Interactive Speeds 2022 SIGMOD 5.2340097e-05
9,126 Modyn: Data-Centric Machine Learning Pipeline Orchestration 2025 SIGMOD 5.2238659e-05
9,256 DQDF: Data-Quality-Aware Dataframes 2022 VLDB 5.2032182e-05
9,426 Towards Observability for Production Machine Learning Pipelines 2022 VLDB 5.1779092e-05
10,185 In-Database Data Imputation 2024 SIGMOD 5.0629036e-05
10,841 PrepBench: How Far Are We from Natural-Language-Driven Data Preparation? 2026 VLDB 4.9769913e-05
10,954 Morphing-based Compression for Data-centric ML Pipelines 2026 VLDB 4.9769913e-05
11,292 CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines 2025 VLDB 4.9769913e-05
11,448 T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data 2025 VLDB 4.9769913e-05
11,595 Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines 2024 VLDB 4.9769913e-05
11,795 CM-Explorer: Dissecting Data Ingestion Problems 2023 VLDB 4.9769913e-05
11,831 Data Management Opportunities for Foundation Models 2022 CIDR 4.9769913e-05
13,820 DEEM 2019: Workshop on Data Management for End-to-End Machine Learning 2019 SIGMOD -
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
23 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00055384955
83 Space-Efficient Online Computation of Quantile Summaries 2001 SIGMOD 0.00035962466
104 HoloClean: Holistic Data Repairs with Probabilistic Inference 2017 VLDB 0.00033676943
204 Declarative Data Cleaning: Language, Model, and Algorithms 2001 VLDB 0.00025179068
350 Discovering Denial Constraints 2013 VLDB 0.00020244085
435 Mining Database Structure; Or, How to Build a Data Quality Browser 2002 SIGMOD 0.0001831946
509 Goods: Organizing Google's Datasets 2016 SIGMOD 0.00017063491
603 A Hybrid Approach to Functional Dependency Discovery 2016 SIGMOD 0.00015692059
716 Guided Data Repair 2011 VLDB 0.00014546968
1,044 Data Cleaning: Overview and Emerging Challenges 2016 SIGMOD 0.00012329478
1,153 Data Management Challenges in Production Machine Learning 2017 SIGMOD 0.00011793347
1,537 Cardinality Estimation: An Experimental Survey 2018 VLDB 0.00010324934
1,881 Ground: A Data Context Service 2017 CIDR 9.4372419e-05
2,447 noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts 2017 VLDB 8.4530494e-05
5,308 Probabilistic Demand Forecasting at Scale 2017 VLDB 6.1844285e-05
5,882 ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning 2016 SIGMOD 5.9560519e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 12,151 Query-Driven Learning for Next Generation Predictive Modeling & Analytics 2019 SIGMOD
2 10,902 Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation 2026 VLDB
3 2,636 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD
4 650 Discovering Data Quality Rules 2008 VLDB
5 5,000 Descriptive and Prescriptive Data Cleaning 2014 SIGMOD
6 9,239 Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables 2025 SIGMOD
7 13,113 Systematic Development of Data Mining-Based Data Quality Tools 2003 VLDB
8 9,256 DQDF: Data-Quality-Aware Dataframes 2022 VLDB
9 7,742 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD
10 6,900 Unit Testing Data with Deequ 2019 SIGMOD