Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors
Summary: Audit of BIRD and Spider 2.0‑Snow finds 52.8% and 66.1% annotation errors (wrong gold SQLs, ambiguity), invalidating much benchmark signal. Re-evaluation of five models shows −3% to +31% shifts and up to three-rank changes, demanding higher-quality benchmarks and improved annotation pipelines. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Tengjun Jin (University of Illinois Urbana-Champaign)
- 2. Yoojin Choi (University of Illinois Urbana-Champaign)
- 3. Yuxuan Zhu (University of Illinois Urbana-Champaign)
- 4. Daniel Kang (University of Illinois Urbana-Champaign)
BibTeX Citation
@inproceedings{jin_cidr26,
address = {Amsterdam, Netherlands},
series = {{CIDR} '26},
title = {{Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors}},
booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
author = {Jin, Tengjun and Choi, Yoojin and Zhu, Yuxuan and Kang, Daniel},
year = {2026}
}Incoming Citations (Sorted by Pagerank)
Showing 1 of 1 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 10,940 | Bridging NL2SQL for CAN Signal Analytics at NIO: From Externalized Schemas to CTE Pipelines | 2026 | VLDB | 4.9793485e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 6 of 6 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 46 | Benchmarking Database Systems: A Systematic Approach | 1983 | VLDB | 0.00044784799 |
| 174 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB | 0.00026790979 |
| 479 | The Making of TPC-DS | 2006 | VLDB | 0.00017622471 |
| 1,777 | OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale | 2025 | VLDB | 9.664552e-05 |
| 2,025 | OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment | 2025 | SIGMOD | 9.163622e-05 |
| 2,189 | A Methodology for Database System Performance Evaluation | 1984 | SIGMOD | 8.8869065e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 1,866 | ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems | 2024 | VLDB |
| 2 | 9,216 | BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation | 2026 | CIDR |
| 3 | 2,348 | The Dawn of Natural Language to SQL: Are We Fully Ready? | 2024 | VLDB |
| 4 | 5,155 | An In-Depth Benchmarking of Text-to-SQL Systems | 2021 | SIGMOD |
| 5 | 7,012 | Reliable Text-to-SQL with Adaptive Abstention | 2025 | SIGMOD |
| 6 | 5,544 | Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency | 2025 | SIGMOD |
| 7 | 10,695 | NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions | 2026 | VLDB |
| 8 | 174 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB |
| 9 | 10,893 | Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation | 2026 | VLDB |
| 10 | 11,071 | Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards | 2026 | VLDB |