DBScholar

Back to papers

Automating Large-Scale Data Quality Verification

Summary: Automates quality verification with a declarative API that blends standard constraints and user code to enable data unit tests. Translates validation to Spark aggregations, enables checks, and uses ML for constraint suggestions and anomaly detection. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
hb2441161cf78104e
Venue
VLDB
Year
2018
Pagerank
0.00011208469
Overall Rank
1,278 | 91.44%
DOI
10.14778/3229863.3229867
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{schelter_vldb18,
        title = {{Automating Large-Scale Data Quality Verification}},
        author = {Schelter, Sebastian and Lange, Dustin and Schmidt, Philipp and Celikel, Meltem and Biessmann, Felix and Grafberger, Andreas},
        journal = {PVLDB},
        series = {{VLDB} '18},
        volume = {11},
        number = {12},
        pages = {1781--1794},
        doi = {10.14778/3229863.3229867},
        url = {https://doi.org/10.14778/3229863.3229867},
        year = {2018}
}

Incoming Citations (Sorted by Pagerank)

Showing 29 of 29 citing papers.

Rank Citing Paper Year Venue Pagerank
1,616 SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle 2020 CIDR 0.00010048776
1,690 Annotating Columns with Pre-trained Language Models 2022 SIGMOD 9.8426663e-05
2,341 SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging 2021 SIGMOD 8.5984124e-05
2,645 Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks 2020 SIGMOD 8.1641056e-05
2,752 Uni-Detect: A Unified Approach to Automated Error Detection in Tables 2019 SIGMOD 8.0387193e-05
3,711 TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines 2020 SIGMOD 7.0580105e-05
4,690 spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines 2024 VLDB 6.4506346e-05
5,117 Towards Benchmarking Feature Type Inference for AutoML Platforms 2021 SIGMOD 6.2551509e-05
5,230 Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications 2023 SIGMOD 6.2059854e-05
5,383 SchemaPile: A Large Collection of Relational Database Schemas 2024 SIGMOD 6.1379374e-05
6,338 Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines 2021 CIDR 5.7918978e-05
6,679 UPLIFT: Parallelization Strategies for Feature Transformations in Machine Learning Workloads 2022 VLDB 5.6956362e-05
6,911 Unit Testing Data with Deequ 2019 SIGMOD 5.6335281e-05
7,755 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD 5.4424293e-05
7,932 Complaint-Driven Training Data Debugging at Interactive Speeds 2022 SIGMOD 5.4074941e-05
8,420 Deducing Certain Fixes to Graphs 2019 VLDB 5.3198062e-05
9,146 Modyn: Data-Centric Machine Learning Pipeline Orchestration 2025 SIGMOD 5.2065925e-05
9,280 DQDF: Data-Quality-Aware Dataframes 2022 VLDB 5.1860131e-05
9,454 Towards Observability for Production Machine Learning Pipelines 2022 VLDB 5.1607878e-05
9,842 CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines 2025 VLDB 5.1064323e-05
10,218 In-Database Data Imputation 2024 SIGMOD 5.0461625e-05
10,397 FlowPipe: LLM-Enhanced Conditional Generative Flow Networks for Data Preparation Pipeline Construction 2027 SIGMOD 4.9605343e-05
10,902 PrepBench: How Far Are We from Natural-Language-Driven Data Preparation? 2026 VLDB 4.9605343e-05
11,015 Morphing-based Compression for Data-centric ML Pipelines 2026 VLDB 4.9605343e-05
11,500 T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data 2025 VLDB 4.9605343e-05
11,643 Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines 2024 VLDB 4.9605343e-05
11,839 CM-Explorer: Dissecting Data Ingestion Problems 2023 VLDB 4.9605343e-05
11,875 Data Management Opportunities for Foundation Models 2022 CIDR 4.9605343e-05
13,860 DEEM 2019: Workshop on Data Management for End-to-End Machine Learning 2019 SIGMOD -
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
23 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00055523999
82 Space-Efficient Online Computation of Quantile Summaries 2001 SIGMOD 0.00036097627
100 HoloClean: Holistic Data Repairs with Probabilistic Inference 2017 VLDB 0.00033951027
206 Declarative Data Cleaning: Language, Model, and Algorithms 2001 VLDB 0.00025121652
348 Discovering Denial Constraints 2013 VLDB 0.00020216677
435 Mining Database Structure; Or, How to Build a Data Quality Browser 2002 SIGMOD 0.00018277436
512 Goods: Organizing Google's Datasets 2016 SIGMOD 0.00017036842
602 A Hybrid Approach to Functional Dependency Discovery 2016 SIGMOD 0.00015670535
717 Guided Data Repair 2011 VLDB 0.00014531658
989 Data Cleaning: Overview and Emerging Challenges 2016 SIGMOD 0.00012638176
1,151 Data Management Challenges in Production Machine Learning 2017 SIGMOD 0.000117662
1,534 Cardinality Estimation: An Experimental Survey 2018 VLDB 0.00010309919
1,878 Ground: A Data Context Service 2017 CIDR 9.431234e-05
2,448 noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts 2017 VLDB 8.4366124e-05
5,327 Probabilistic Demand Forecasting at Scale 2017 VLDB 6.1639984e-05
5,868 ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning 2016 SIGMOD 5.948547e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 12,194 Query-Driven Learning for Next Generation Predictive Modeling & Analytics 2019 SIGMOD
2 10,963 Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation 2026 VLDB
3 2,644 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD
4 653 Discovering Data Quality Rules 2008 VLDB
5 5,022 Descriptive and Prescriptive Data Cleaning 2014 SIGMOD
6 9,263 Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables 2025 SIGMOD
7 13,154 Systematic Development of Data Mining-Based Data Quality Tools 2003 VLDB
8 9,280 DQDF: Data-Quality-Aware Dataframes 2022 VLDB
9 7,755 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD
10 6,911 Unit Testing Data with Deequ 2019 SIGMOD