DBScholar

Back to papers

Automating Large-Scale Data Quality Verification

Summary: Automates quality verification with a declarative API that blends standard constraints and user code to enable data unit tests. Translates validation to Spark aggregations, enables checks, and uses ML for constraint suggestions and anomaly detection. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
11848
Venue
VLDB
Year
2018
Pagerank
0.00011065626
Overall Rank
1,350 | 90.74%
DOI
10.14778/3229863.3229867

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{schelter_vldb18,
        title = {{Automating Large-Scale Data Quality Verification}},
        author = {Schelter, Sebastian and Lange, Dustin and Schmidt, Philipp and Celikel, Meltem and Biessmann, Felix and Grafberger, Andreas},
        journal = {PVLDB},
        series = {{VLDB} '18},
        volume = {11},
        number = {12},
        pages = {1781--1794},
        doi = {10.14778/3229863.3229867},
        url = {https://doi.org/10.14778/3229863.3229867},
        year = {2018}
}

Incoming Citations (Sorted by Pagerank)

Showing 27 of 27 citing papers.

Rank Citing Paper Year Venue Pagerank
1,756 SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle 2020 CIDR 9.8172465e-05
1,923 Annotating Columns with Pre-trained Language Models 2022 SIGMOD 9.4789109e-05
2,273 SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging 2021 SIGMOD 8.8230899e-05
2,728 Uni-Detect: A Unified Approach to Automated Error Detection in Tables 2019 SIGMOD 8.1995954e-05
2,744 Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks 2020 SIGMOD 8.1781662e-05
3,630 TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines 2020 SIGMOD 7.2387749e-05
4,621 spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines 2024 VLDB 6.6024412e-05
5,051 Towards Benchmarking Feature Type Inference for AutoML Platforms 2021 SIGMOD 6.385354e-05
5,466 SchemaPile: A Large Collection of Relational Database Schemas 2024 SIGMOD 6.2075052e-05
6,250 Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines 2021 CIDR 5.9418015e-05
6,538 UPLIFT: Parallelization Strategies for Feature Transformations in Machine Learning Workloads 2022 VLDB 5.8477764e-05
6,801 Unit Testing Data with Deequ 2019 SIGMOD 5.7690726e-05
7,232 Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications 2023 SIGMOD 5.6659017e-05
7,690 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD 5.5669307e-05
8,225 Deducing Certain Fixes to Graphs 2019 VLDB 5.4624971e-05
8,859 Complaint-Driven Training Data Debugging at Interactive Speeds 2022 SIGMOD 5.356561e-05
8,949 Modyn: Data-Centric Machine Learning Pipeline Orchestration 2025 SIGMOD 5.3462965e-05
9,069 DQDF: Data-Quality-Aware Dataframes 2022 VLDB 5.3251649e-05
9,245 Towards Observability for Production Machine Learning Pipelines 2022 VLDB 5.2992628e-05
9,993 In-Database Data Imputation 2024 SIGMOD 5.1815618e-05
10,589 Morphing-based Compression for Data-centric ML Pipelines 2026 VLDB 5.093636e-05
10,882 CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines 2025 VLDB 5.093636e-05
11,090 T-Assess: An Efficient Data Quality Assessment System Tailored for Trajectory Data 2025 VLDB 5.093636e-05
11,260 Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines 2024 VLDB 5.093636e-05
11,479 CM-Explorer: Dissecting Data Ingestion Problems 2023 VLDB 5.093636e-05
11,516 Data Management Opportunities for Foundation Models 2022 CIDR 5.093636e-05
13,501 DEEM 2019: Workshop on Data Management for End-to-End Machine Learning 2019 SIGMOD -
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 16 of 16 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
24 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00054865648
82 Space-Efficient Online Computation of Quantile Summaries 2001 SIGMOD 0.00036378991
112 HoloClean: Holistic Data Repairs with Probabilistic Inference 2017 VLDB 0.00032801121
201 Declarative Data Cleaning: Language, Model, and Algorithms 2001 VLDB 0.00025558602
376 Discovering Denial Constraints 2013 VLDB 0.00019677674
432 Mining Database Structure; Or, How to Build a Data Quality Browser 2002 SIGMOD 0.00018572055
514 Goods: Organizing Google's Datasets 2016 SIGMOD 0.00017178673
618 A Hybrid Approach to Functional Dependency Discovery 2016 SIGMOD 0.00015711835
714 Guided Data Repair 2011 VLDB 0.00014662041
1,147 Data Management Challenges in Production Machine Learning 2017 SIGMOD 0.00011974846
1,323 Data Cleaning: Overview and Emerging Challenges 2016 SIGMOD 0.00011152602
1,516 Cardinality Estimation: An Experimental Survey 2018 VLDB 0.00010520885
1,902 Ground: A Data Context Service 2017 CIDR 9.506714e-05
2,431 noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts 2017 VLDB 8.5875984e-05
5,182 Probabilistic Demand Forecasting at Scale 2017 VLDB 6.3287692e-05
5,794 ActiveClean: An Interactive Data Cleaning Framework For Modern Machine Learning 2016 SIGMOD 6.0863045e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 1,323 Data Cleaning: Overview and Emerging Challenges 2016 SIGMOD
2 11,845 Query-Driven Learning for Next Generation Predictive Modeling & Analytics 2019 SIGMOD
3 2,594 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD
4 647 Discovering Data Quality Rules 2008 VLDB
5 4,890 Descriptive and Prescriptive Data Cleaning 2014 SIGMOD
6 10,785 Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables 2025 SIGMOD
7 12,817 Systematic Development of Data Mining-Based Data Quality Tools 2003 VLDB
8 9,069 DQDF: Data-Quality-Aware Dataframes 2022 VLDB
9 7,690 Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes 2021 SIGMOD
10 6,801 Unit Testing Data with Deequ 2019 SIGMOD