Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards
Summary: Audits BIRD and Spider 2.0-Snow, finding annotation error rates of 52.8% and 62.8%. Correcting BIRD labels changes agent scores by −7–31% and rankings by up to nine places, exposing unreliable text-to-SQL leaderboards. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Tengjun Jin (University of Illinois Urbana-Champaign)
- 2. Yoojin Choi (University of Illinois Urbana-Champaign)
- 3. Yuxuan Zhu (University of Illinois Urbana-Champaign)
- 4. Daniel Kang (University of Illinois Urbana-Champaign)
BibTeX Citation
@article{jin_vldb26,
title = {{Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards}},
author = {Jin, Tengjun and Choi, Yoojin and Zhu, Yuxuan and Kang, Daniel},
journal = {PVLDB},
series = {{VLDB} '26},
volume = {19},
number = {5},
pages = {931--944},
doi = {10.14778/3796195.3796206},
url = {https://doi.org/10.14778/3796195.3796206},
year = {2026}
}Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 4 of 4 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 279 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB | 0.00022468369 |
| 756 | CodeS: Towards Building Open-source Language Models for Text-to-SQL | 2024 | SIGMOD | 0.0001431656 |
| 2,395 | OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale | 2025 | VLDB | 8.6355093e-05 |
| 2,710 | OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment | 2025 | SIGMOD | 8.2206448e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 10,217 | DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework | 2026 | SIGMOD |
| 2 | 10,556 | OpenSQL: Data-Efficient Text-to-SQL for Open-Source LLMs via Synthesized Intermediate Supervision | 2026 | VLDB |
| 3 | 5,174 | An In-Depth Benchmarking of Text-to-SQL Systems | 2021 | SIGMOD |
| 4 | 2,852 | The Dawn of Natural Language to SQL: Are We Fully Ready? | 2024 | VLDB |
| 5 | 6,267 | Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency | 2025 | SIGMOD |
| 6 | 10,122 | BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation | 2026 | CIDR |
| 7 | 6,936 | Reliable Text-to-SQL with Adaptive Abstention | 2025 | SIGMOD |
| 8 | 279 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB |
| 9 | 10,510 | NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions | 2026 | VLDB |
| 10 | 10,141 | Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors | 2026 | CIDR |