SQLStorm: Taking Database Benchmarking into the LLM Era

Summary: LLM-driven methodology generates SQLStorm: 18K+ realistic queries across 1–220 GB for only $15. Broad SQL coverage beyond TPC-H/DS/JOB enables compatibility testing, bug discovery, optimizer/cardinality-estimation research, and robust performance evaluation. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: h12a91e8aa8604d4f
Venue: VLDB
Year: 2025
Pagerank: 6.9127576e-05
Overall Rank: 3,909 | 73.80%
DOI: 10.14778/3749646.3749683
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Tobias Schmidt (Technical University of Munich)
2. Viktor Leis (Technical University of Munich)
3. Peter Boncz (Centrum Wiskunde & Informatica)
4. Thomas Neumann (Technical University of Munich)

BibTeX Citation

@article{schmidt_vldb25,
        title = {{SQLStorm: Taking Database Benchmarking into the LLM Era}},
        author = {Schmidt, Tobias and Leis, Viktor and Boncz, Peter and Neumann, Thomas},
        journal = {PVLDB},
        series = {{VLDB} '25},
        volume = {18},
        number = {11},
        pages = {4144--4157},
        doi = {10.14778/3749646.3749683},
        url = {https://doi.org/10.14778/3749646.3749683},
        year = {2025}
}

Incoming Citations (Sorted by Pagerank)

Showing 17 of 17 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,158	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD	7.5676704e-05
9,253	Finding Missed Optimizations in DBMSs through Unbalanced Short-Circuit Query Construction	2026	SIGMOD	5.1860131e-05
9,494	SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads	2026	SIGMOD	5.1541298e-05
10,143	Still Asking: How Good Are Query Optimizers, Really?	2025	VLDB	5.0597449e-05
10,278	Redbench: Workload Synthesis From Cloud Traces	2026	VLDB	5.0334833e-05
10,391	DQSim: Does the Task Assignment Slow Down Your Distributed Database System?	2027	SIGMOD	4.9605343e-05
10,420	Rethinking Query Optimization for Multi-Agent Systems	2027	VLDB	4.9605343e-05
10,436	On the Vexing Difficulty of Evaluating IN Predicates	2026	CIDR	4.9605343e-05
10,493	Automated Discovery of Test Oracles for Database Management Systems Using LLMs	2026	SIGMOD	4.9605343e-05
10,514	Dialect-Agnostic SQL Parsing via LLM-Based Segmentation	2026	SIGMOD	4.9605343e-05
10,699	CorrBound: Cardinality Estimation Accounting for Inter- and Intra-relation Correlations	2026	SIGMOD	4.9605343e-05
10,784	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	4.9605343e-05
10,822	Toward Drift-Aware Database Benchmarking	2026	VLDB	4.9605343e-05
10,903	ReSequel: Robust LLM-assisted Query Rewriting and Optimization using Templatization and Sampling	2026	VLDB	4.9605343e-05
10,956	Accelerating String-Heavy Queries with LLM Token Tables	2026	VLDB	4.9605343e-05
11,025	How Out-of-Bounds Are Your Cardinality Estimates?	2026	VLDB	4.9605343e-05
11,066	Persona-Conditioned Query Generation for Selectivity-Controlled Workloads	2026	VLDB	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 31 of 31 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
15	How Good Are Query Optimizers, Really?	2016	VLDB	0.00061463175
84	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035920657
323	Can Foundation Models Wrangle Your Data?	2023	VLDB	0.00021022156
355	Bao: Making Learned Query Optimization Practical	2021	SIGMOD	0.00020146245
381	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019497061
983	Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation	2022	VLDB	0.00012677179
1,238	DB-BERT: A Database Tuning Tool that "Reads the Manual"	2022	SIGMOD	0.00011375179
1,513	DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems	2021	VLDB	0.00010394557
1,656	D-Bot: Database Diagnosis System using Large Language Models	2024	VLDB	9.948529e-05
1,791	Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet	2024	VLDB	9.6212919e-05
1,931	CodexDB: Synthesizing Code for Query Processing from Natural Language Instructions using GPT-3 Codex	2022	VLDB	9.3449634e-05
1,956	LLM-R^2: A Large Language Model Enhanced Rule-based Rewrite System for Boosting Query Efficiency	2025	VLDB	9.2927535e-05
1,958	Quantifying TPC-H Choke Points and Their Optimizations	2020	VLDB	9.2894133e-05
2,227	GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization	2024	VLDB	8.7924681e-05
3,158	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD	7.5676704e-05
3,303	Cloud Analytics Benchmark	2023	VLDB	7.4212268e-05
3,361	Exact Cardinality Query Optimization for Optimizer Testing	2009	VLDB	7.3621595e-05
3,474	DIAMetrics: Benchmarking Query Engines at Scale	2020	VLDB	7.256029e-05
3,839	Automatic Database Configuration Debugging using Retrieval-Augmented Language Models	2025	SIGMOD	6.9704205e-05
4,351	Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs	2024	VLDB	6.634024e-05
4,383	R-Bot: An LLM-based Query Rewrite System	2025	VLDB	6.6140223e-05
4,729	LLM for Data Management	2024	VLDB	6.4317927e-05
4,971	Debunking the Myth of Join Ordering: Toward Robust SQL Analytics	2025	SIGMOD	6.3209003e-05
5,044	Analyzing the Impact of Cardinality Estimation on Execution Plans in Microsoft SQL Server	2023	VLDB	6.2888728e-05
5,911	Yannakakis+: Practical Acyclic Query Evaluation with Theoretical Guarantees	2025	SIGMOD	5.9338021e-05
5,925	Join Size Bounds using l_p-Norms on Degree Sequences	2024	PODS	5.9282898e-05
6,584	Can Large Language Models Be Query Optimizer for Relational Databases?	2026	SIGMOD	5.72395e-05
6,658	Workload Insights From The Snowflake Data Cloud: What Do Production Analytic Queries Really Look Like?	2025	VLDB	5.7025639e-05
7,827	Parachute: Single-Pass Bi-Directional Information Passing	2025	VLDB	5.4297836e-05
9,003	DataLoom: Simplifying Data Loading with LLMs	2024	VLDB	5.2256178e-05
9,662	Low Rank Learning for Offline Query Optimization	2025	SIGMOD	5.1283524e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	10,963	Developing and Benchmarking Verification Algorithms to Improve Text-to-SQL Generation	2026	VLDB
2	9,250	BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation	2026	CIDR
3	5,111	An In-Depth Benchmarking of Text-to-SQL Systems	2021	SIGMOD
4	11,043	Demonstrating GenDB: Instance-Optimized and Customized Query Processing Code Generation via LLM Agents	2026	VLDB
5	10,323	This is Going to Sound Crazy, But What If We Used Large Language Models to Boost Automatic Database Tuning Algorithms By Leveraging Prior History? We Will Find Better Configurations More Quickly Than Retraining From Scratch!	2026	SIGMOD
6	10,766	NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions	2026	VLDB
7	9,754	Leveraging Query Optimizers to Verify the Soundness of LLM-based Query Rewrites for Real-World Workloads, and More!	2026	CIDR
8	9,494	SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads	2026	SIGMOD
9	3,158	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD
10	163	Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation	2024	VLDB