Learning to Validate the Predictions of Black Box Classifiers on Unseen Data

Summary: Learns a performance predictor for pretrained black-box classifiers using programmatic specifications of dataset shift and data errors, without distributional assumptions. Alarms on predicted accuracy drops on unseen serving data and outperforms baselines across datasets and error types. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 5879
Venue: SIGMOD
Year: 2020
Pagerank: 7.2118928e-05
Overall Rank: 3,670 | 74.83%
DOI: 10.1145/3318464.3380604

Incoming Non-self Citations Over Time

Authors

1. Sebastian Schelter (New York University)
2. Tammo Rukat (Amazon)
3. Felix Biessmann (Beuth University Berlin)

BibTeX Citation

@inproceedings{schelter_sigmod20,
        title = {{Learning to Validate the Predictions of Black Box Classifiers on Unseen Data}},
        author = {Schelter, Sebastian and Rukat, Tammo and Biessmann, Felix},
        series = {{SIGMOD} '20},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3318464.3380604},
        url = {https://dl.acm.org/doi/10.1145/3318464.3380604},
        year = {2020}
}

Incoming Citations (Sorted by Pagerank)

Showing 10 of 10 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,273	SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging	2021	SIGMOD	8.8230899e-05
4,835	Adaptive Data Augmentation for Supervised Learning over Missing Data	2021	VLDB	6.486592e-05
7,068	DataPrism: Exposing Disconnect between Data and Systems	2022	SIGMOD	5.7119157e-05
7,232	Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications	2023	SIGMOD	5.6659017e-05
7,397	Conformance Constraint Discovery: Measuring Trust in Data-Driven Systems	2021	SIGMOD	5.6256632e-05
8,949	Modyn: Data-Centric Machine Learning Pipeline Orchestration	2025	SIGMOD	5.3462965e-05
9,243	Scapin: Scalable Graph Structure Perturbation by Augmented Influence Maximization	2023	SIGMOD	5.2993405e-05
9,881	The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format	2024	SIGMOD	5.2040783e-05
11,260	Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines	2024	VLDB	5.093636e-05
11,698	Comprehensible Counterfactual Explanation on Kolmogorov-Smirnov Test	2021	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 1 of 1 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
2,728	Uni-Detect: A Unified Approach to Automated Error Detection in Tables	2019	SIGMOD	8.1995954e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,630	TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines	2020	SIGMOD
2	11,512	Towards Observability for Machine Learning Pipelines	2022	CIDR
3	10,298	The Power of Anomaly Detection in Predictive Maintenance: [Experiments & Analysis]	2026	SIGMOD
4	7,690	Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes	2021	SIGMOD
5	2,313	Active Learning for ML Enhanced Database Systems	2020	SIGMOD
6	1,350	Automating Large-Scale Data Quality Verification	2018	VLDB
7	11,211	MisDetect: Iterative Mislabel Detection using Early Loss	2024	VLDB
8	6,431	Finding Label and Model Errors in Perception Data With Learned Observation Assertions	2022	SIGMOD
9	9,245	Towards Observability for Production Machine Learning Pipelines	2022	VLDB
10	11,260	Efficiently Mitigating the Impact of Data Drift on Machine Learning Pipelines	2024	VLDB