An In-Depth Benchmarking of Text-to-SQL Systems

Summary: Rigorous, multi-class Text-to-SQL benchmark covering diverse query types beyond existing datasets. Systematic evaluation of several T2SQL systems with execution time and resource usage, revealing gaps, capabilities, and open challenges in current approaches. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6178
Venue: SIGMOD
Year: 2021
Pagerank: 6.3337874e-05
Overall Rank: 5,174 | 64.51%
DOI: 10.1145/3448016.3452836

Incoming Non-self Citations Over Time

Authors

1. Orest Gkini (Athena Research Center)
2. Theofilos Belmpas (Athena Research Center)
3. Georgia Koutrika (Athena Research Center)
4. Yannis Ioannidis (Athena Research Center; University of Athens)

BibTeX Citation

@inproceedings{gkini_sigmod21,
        title = {{An In-Depth Benchmarking of Text-to-SQL Systems}},
        author = {Gkini, Orest and Belmpas, Theofilos and Koutrika, Georgia and Ioannidis, Yannis},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3452836},
        url = {https://dl.acm.org/doi/10.1145/3448016.3452836},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,852	The Dawn of Natural Language to SQL: Are We Fully Ready?	2024	VLDB	8.0455088e-05
3,339	A Deep Dive into Deep Learning Approaches for Text-to-SQL Systems	2021	SIGMOD	7.5047676e-05
8,945	Generation of Training Examples for Tabular Natural Language Inference	2023	SIGMOD	5.3480421e-05
9,136	Sphinteract: Resolving Ambiguities in NL2SQL Through User Interaction	2025	VLDB	5.3166292e-05
9,305	The Power of Constraints in Natural Language to SQL Translation	2025	VLDB	5.289545e-05

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
37	DISCOVER: Keyword Search in Relational Databases	2002	VLDB	0.00048017193
179	Constructing an Interactive Natural Language Interface for Relational Databases	2015	VLDB	0.00026838277
212	Efficient IR-Style Keyword Search over Relational Databases	2003	VLDB	0.000247733
272	BLINKS: Ranked Keyword Searches on Graphs	2007	SIGMOD	0.00022695855
459	ATHENA: An Ontology-Driven System for Natural Language Querying over Relational Data Stores	2016	VLDB	0.00018101318
930	SPARK: Top-k Keyword Query in Relational Databases	2007	SIGMOD	0.0001312728
1,182	SQAK: Doing More with Keywords	2008	SIGMOD	0.00011778765
1,214	SODA: Generating SQL for Business Users	2012	VLDB	0.0001163751
5,130	Natural Language Data Management and Interfaces: Recent Development and Open Challenges	2017	SIGMOD	6.354484e-05
6,567	MeanKS: Meaningful Keyword Search in Relational Databases with Complex Schema	2014	SIGMOD	5.8380657e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	10,732	RTS+: Reliable Text to SQL	2025	SIGMOD
2	45	Benchmarking Database Systems: A Systematic Approach	1983	VLDB
3	10,406	Test Data Generation for Complex SQL Queries	2026	SIGMOD
4	10,537	TACO: A Benchmark for Open-Domain Text-to-SQL with Ambiguous and Cross-Database Queries	2026	VLDB
5	3,339	A Deep Dive into Deep Learning Approaches for Text-to-SQL Systems	2021	SIGMOD
6	10,510	NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions	2026	VLDB
7	7,071	The TEXTURE Benchmark: Measuring Performance of Text Queries on a Relational DBMS	2005	VLDB
8	865	Natural language to SQL: Where are we today?	2020	VLDB
9	279	Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation	2024	VLDB
10	10,211	Comparison and Analysis of Value Linking in Text-to-SQL Systems [Experiments & Analysis]	2026	SIGMOD