Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors
Summary: Audit of BIRD and Spider 2.0‑Snow finds 52.8% and 66.1% annotation errors (wrong gold SQLs, ambiguity), invalidating much benchmark signal. Re-evaluation of five models shows −3% to +31% shifts and up to three-rank changes, demanding higher-quality benchmarks and improved annotation pipelines. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Tengjun Jin
- 2. Yoojin Choi
- 3. Yuxuan Zhu
- 4. Daniel Kang
Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 6 of 6 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 10 | Benchmarking Database Systems: A Systematic Approach | 1983 | VLDB | 0.0012103754 |
| 369 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB | 0.0002547515 |
| 659 | The Making of TPC-DS | 2006 | VLDB | 0.00018500853 |
| 1,965 | A Methodology for Database System Performance Evaluation | 1984 | SIGMOD | 9.918599e-05 |
| 3,859 | OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment | 2025 | SIGMOD | 6.6907933e-05 |
| 3,978 | OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale | 2025 | VLDB | 6.5725884e-05 |
Previous
Page 1 / 1
Next