Dynamic Speculative Optimizations for SQL Compilation in Apache Spark

Summary: Dynamic speculative optimizations for Spark SQL compilation via runtime profiling and adaptive codegen to reduce data access and deserialization overhead on textual formats. Achieves up to 4.4x speedups on TPC-H with CSV/JSON, illustrating a unique runtime-driven codegen approach for Spark. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h1efd1cc1aea056bc
Venue: VLDB
Year: 2020
Pagerank: 5.1943472e-05
Overall Rank: 9,321 | 37.34%
DOI: 10.14778/3377369.3377382

Incoming Non-self Citations Over Time

Authors

1. Filippo Schiavio (Università della Svizzera italiana)
2. Daniele Bonetta (Oracle)
3. Walter Binder (Università della Svizzera italiana)

BibTeX Citation

@article{schiavio_vldb20,
        title = {{Dynamic Speculative Optimizations for SQL Compilation in Apache Spark}},
        author = {Schiavio, Filippo and Bonetta, Daniele and Binder, Walter},
        journal = {PVLDB},
        series = {{VLDB} '20},
        volume = {13},
        number = {5},
        pages = {754--767},
        doi = {10.14778/3377369.3377382},
        url = {https://doi.org/10.14778/3377369.3377382},
        year = {2020}
}

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,793	Permutable Compiled Queries: Dynamically Adapting Compiled Queries without Recompiling	2021	VLDB	6.4127583e-05
8,673	Adaptive Code Generation for Data-Intensive Analytics	2021	VLDB	5.2913671e-05
9,600	Language-Agnostic Integrated Queries in a Managed Polyglot Runtime	2021	VLDB	5.1548991e-05

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
21	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00056855599
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055406774
605	Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask	2018	VLDB	0.00015647561
1,341	Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last	2018	VLDB	0.00010968512
2,347	Filter Before You Parse: Faster Analytics on Raw Data with Sparser	2018	VLDB	8.6012176e-05
2,436	Mison: A Fast JSON Parser for Data Analytics	2017	VLDB	8.4708399e-05
2,618	The MemSQL Query Optimizer: A modern optimizer for real-time analytics in a distributed database	2016	VLDB	8.2217101e-05
3,455	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	7.2884813e-05
4,069	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	6.821366e-05
5,140	FAD.js: Fast JSON Data Access Using JIT-based Speculative Optimizations	2017	VLDB	6.2575221e-05
6,278	Just-In-Time Data Virtualization: Lightweight Data Management with ViDa	2015	CIDR	5.8263939e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,123	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
2	3,455	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
3	11,443	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB
4	8,346	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB
5	23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD
6	8,535	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB
7	2,735	Speculative Distributed CSV Data Parsing for Big Data Analytics	2019	SIGMOD
8	10,887	MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning	2026	VLDB
9	2,635	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
10	8,004	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB