SQLStorm: Taking Database Benchmarking into the LLM Era

Summary: LLM-driven methodology generates SQLStorm: 18K+ realistic queries across 1–220 GB for only $15. Broad SQL coverage beyond TPC-H/DS/JOB enables compatibility testing, bug discovery, optimizer/cardinality-estimation research, and robust performance evaluation. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 14221
Venue: VLDB
Year: 2025
Pagerank: 6.1019672e-05
Overall Rank: 5,744 | 60.60%
DOI: 10.14778/3749646.3749683

Incoming Non-self Citations Over Time

Authors

1. Tobias Schmidt (Technical University of Munich)
2. Viktor Leis (Technical University of Munich)
3. Peter Boncz (Centrum Wiskunde & Informatica)
4. Thomas Neumann (Technical University of Munich)

BibTeX Citation

@article{schmidt_vldb25,
        title = {{SQLStorm: Taking Database Benchmarking into the LLM Era}},
        author = {Schmidt, Tobias and Leis, Viktor and Boncz, Peter and Neumann, Thomas},
        journal = {PVLDB},
        series = {{VLDB} '25},
        volume = {18},
        number = {11},
        pages = {4144--4157},
        doi = {10.14778/3749646.3749683},
        url = {https://doi.org/10.14778/3749646.3749683},
        year = {2025}
}

Incoming Citations (Sorted by Pagerank)

Showing 10 of 10 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,363	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD	6.7423909e-05
9,920	Still Asking: How Good Are Query Optimizers, Really?	2025	VLDB	5.1955087e-05
10,136	On the Vexing Difficulty of Evaluating IN Predicates	2026	CIDR	5.093636e-05
10,199	Automated Discovery of Test Oracles for Database Management Systems Using LLMs	2026	SIGMOD	5.093636e-05
10,220	Dialect-Agnostic SQL Parsing via LLM-Based Segmentation	2026	SIGMOD	5.093636e-05
10,243	Finding Missed Optimizations in DBMSs through Unbalanced Short-Circuit Query Construction	2026	SIGMOD	5.093636e-05
10,438	CorrBound: Cardinality Estimation Accounting for Inter- and Intra-relation Correlations	2026	SIGMOD	5.093636e-05
10,501	SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads	2026	SIGMOD	5.093636e-05
10,529	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	5.093636e-05
10,569	Toward Drift-Aware Database Benchmarking	2026	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 31 of 31 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
18	How Good Are Query Optimizers, Really?	2016	VLDB	0.00059284255
84	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035838391
378	Bao: Making Learned Query Optimization Practical	2021	SIGMOD	0.00019638121
388	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019410042
420	Can Foundation Models Wrangle Your Data?	2023	VLDB	0.00018789852
1,122	Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation	2022	VLDB	0.0001209124
1,337	DB-BERT: A Database Tuning Tool that "Reads the Manual"	2022	SIGMOD	0.00011117488
1,832	DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems	2021	VLDB	9.6607418e-05
1,920	D-Bot: Database Diagnosis System using Large Language Models	2024	VLDB	9.4846185e-05
2,202	Quantifying TPC-H Choke Points and Their Optimizations	2020	VLDB	8.9639459e-05
2,298	GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization	2024	VLDB	8.7886538e-05
2,408	Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet	2024	VLDB	8.6154404e-05
2,521	CodexDB: Synthesizing Code for Query Processing from Natural Language Instructions using GPT-3 Codex	2022	VLDB	8.4729505e-05
2,553	LLM-R^2: A Large Language Model Enhanced Rule-based Rewrite System for Boosting Query Efficiency	2025	VLDB	8.4283807e-05
3,426	Exact Cardinality Query Optimization for Optimizer Testing	2009	VLDB	7.4218997e-05
3,460	DIAMetrics: Benchmarking Query Engines at Scale	2020	VLDB	7.3939262e-05
3,838	Cloud Analytics Benchmark	2023	VLDB	7.0823175e-05
4,363	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD	6.7423909e-05
4,886	Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs	2024	VLDB	6.4614955e-05
5,010	Analyzing the Impact of Cardinality Estimation on Execution Plans in Microsoft SQL Server	2023	VLDB	6.4023732e-05
5,529	Debunking the Myth of Join Ordering: Toward Robust SQL Analytics	2025	SIGMOD	6.18591e-05
5,856	Automatic Database Configuration Debugging using Retrieval-Augmented Language Models	2025	SIGMOD	6.0644656e-05
6,101	LLM for Data Management	2024	VLDB	5.9774273e-05
6,257	Join Size Bounds using l_p-Norms on Degree Sequences	2024	PODS	5.9397944e-05
6,259	R-Bot: An LLM-based Query Rewrite System	2025	VLDB	5.93967e-05
6,434	Yannakakis+: Practical Acyclic Query Evaluation with Theoretical Guarantees	2025	SIGMOD	5.8799421e-05
7,809	Can Large Language Models Be Query Optimizer for Relational Databases?	2026	SIGMOD	5.5399022e-05
8,074	Parachute: Single-Pass Bi-Directional Information Passing	2025	VLDB	5.4929961e-05
8,846	Workload Insights From The Snowflake Data Cloud: What Do Production Analytic Queries Really Look Like?	2025	VLDB	5.3580184e-05
8,903	DataLoom: Simplifying Data Loading with LLMs	2024	VLDB	5.3486551e-05
9,601	Low Rank Learning for Offline Query Optimization	2025	SIGMOD	5.2487799e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,045	Logical and Physical Optimizations for SQL Query Execution over Large Language Models	2025	SIGMOD
2	4,944	Hybrid Querying Over Relational Databases and Large Language Models	2025	CIDR
3	10,122	BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation	2026	CIDR
4	5,174	An In-Depth Benchmarking of Text-to-SQL Systems	2021	SIGMOD
5	10,506	This is Going to Sound Crazy, But What If We Used Large Language Models to Boost Automatic Database Tuning Algorithms By Leveraging Prior History? We Will Find Better Configurations More Quickly Than Retraining From Scratch!	2026	SIGMOD
6	10,510	NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions	2026	VLDB
7	10,139	Leveraging Query Optimizers to Verify the Soundness of LLM-based Query Rewrites for Real-World Workloads, and More!	2026	CIDR
8	10,501	SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads	2026	SIGMOD
9	4,363	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD
10	279	Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation	2024	VLDB