spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines
Summary: spade synthesizes data-quality assertions for LLM pipelines by mining prompt-version histories to generate candidate assertion functions and selecting a minimal set meeting coverage and accuracy constraints. Yields fewer assertions and ~21% fewer false failures; deployed in LangSmith. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Shreya Shankar (University of California Berkeley)
- 2. Haotian Li (Hong Kong University of Science and Technology)
- 3. Parth Asawa (University of California Berkeley)
- 4. Madelon Hulsebos (University of California Berkeley)
- 5. Yiming Lin (University of California Berkeley)
- 6. J.D. Zamfirescu-Pereira (University of California Berkeley)
- 7. Harrison Chase (LangChain)
- 8. Will Fu-Hinthorn (LangChain)
- 9. Aditya G. Parameswaran (University of California Berkeley)
- 10. Eugene Wu (Columbia University)
BibTeX Citation
@article{shankar_vldb24,
title = {{spade: Synthesizing Data Quality Assertions for Large Language Model Pipelines}},
author = {Shankar, Shreya and Li, Haotian and Asawa, Parth and Hulsebos, Madelon and Lin, Yiming and Zamfirescu-Pereira, J.D. and Chase, Harrison and Fu-Hinthorn, Will and Parameswaran, Aditya G. and Wu, Eugene},
journal = {PVLDB},
series = {{VLDB} '24},
volume = {17},
number = {12},
pages = {4173--4186},
doi = {10.14778/3685800.3685835},
url = {https://doi.org/10.14778/3685800.3685835},
year = {2024}
}
Incoming Citations (Sorted by Pagerank)
Showing 6 of 6 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,343 | DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing | 2025 | VLDB | 0.00011095866 |
| 9,841 | Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems | 2025 | VLDB | 5.2101877e-05 |
| 10,132 | Making Prompts First-Class Citizens for Adaptive LLM Pipelines | 2026 | CIDR | 5.093636e-05 |
| 10,713 | D-Bot: An LLM-Powered DBA Copilot | 2025 | SIGMOD | 5.093636e-05 |
| 10,909 | LLMLog: Advanced Log Template Generation via LLM-driven Multi-Round Annotation | 2025 | VLDB | 5.093636e-05 |
| 13,311 | Prompt Editor: A Taxonomy-driven System for Guided LLM Prompt Development in Enterprise Settings | 2025 | SIGMOD | - |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 8 of 8 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 205 | Snorkel: Rapid Training Data Creation with Weak Supervision | 2018 | VLDB | 0.00025235185 |
| 420 | Can Foundation Models Wrangle Your Data? | 2023 | VLDB | 0.00018789852 |
| 1,350 | Automating Large-Scale Data Quality Verification | 2018 | VLDB | 0.00011065626 |
| 3,536 | How Large Language Models Will Disrupt Data Management | 2023 | VLDB | 7.3297343e-05 |
| 4,050 | Revisiting Prompt Engineering via Declarative Crowdsourcing | 2024 | CIDR | 6.9368666e-05 |
| 4,592 | Data Platform for Machine Learning | 2019 | SIGMOD | 6.6139071e-05 |
| 6,431 | Finding Label and Model Errors in Perception Data With Learned Observation Assertions | 2022 | SIGMOD | 5.8802622e-05 |
| 7,609 | Ease.ml/ci and Ease.ml/meter in Action: Towards Data Management for Statistical Generalization | 2019 | VLDB | 5.5841767e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 7,439 | AOP: Automated and Interactive LLM Pipeline Orchestration for Answering Complex Queries | 2025 | CIDR |
| 2 | 7,316 | SpareLLM: Automatically Selecting Task-Specific Minimum-Cost Large Language Models under Equivalence Constraint | 2025 | SIGMOD |
| 3 | 10,202 | BAT: Target-Instance-Free Data Preparation Synthesis via LLM-Driven Tree Search | 2026 | SIGMOD |
| 4 | 4,050 | Revisiting Prompt Engineering via Declarative Crowdsourcing | 2024 | CIDR |
| 5 | 11,120 | Welding Natural Language Queries to Analytics IRs with LLMs | 2024 | CIDR |
| 6 | 6,765 | AquaPipe: A Quality-Aware Pipeline for Knowledge Retrieval and Large Language Models | 2025 | SIGMOD |
| 7 | 1,350 | Automating Large-Scale Data Quality Verification | 2018 | VLDB |
| 8 | 10,882 | CatDB: Data-catalog-guided, LLM-based Generation of Data-centric ML Pipelines | 2025 | VLDB |
| 9 | 8,829 | Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees | 2026 | SIGMOD |
| 10 | 10,132 | Making Prompts First-Class Citizens for Adaptive LLM Pipelines | 2026 | CIDR |