Dynamic Speculative Optimizations for SQL Compilation in Apache Spark

Summary: Dynamic speculative optimizations for Spark SQL compilation via runtime profiling and adaptive codegen to reduce data access and deserialization overhead on textual formats. Achieves up to 4.4x speedups on TPC-H with CSV/JSON, illustrating a unique runtime-driven codegen approach for Spark. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12460
Venue: VLDB
Year: 2020
Pagerank: 5.3099969e-05
Overall Rank: 9,167 | 37.11%
DOI: 10.14778/3377369.3377382

Incoming Non-self Citations Over Time

Authors

1. Filippo Schiavio (Università della Svizzera italiana)
2. Daniele Bonetta (Oracle)
3. Walter Binder (Università della Svizzera italiana)

BibTeX Citation

@article{schiavio_vldb20,
        title = {{Dynamic Speculative Optimizations for SQL Compilation in Apache Spark}},
        author = {Schiavio, Filippo and Bonetta, Daniele and Binder, Walter},
        journal = {PVLDB},
        series = {{VLDB} '20},
        volume = {13},
        number = {5},
        pages = {754--767},
        doi = {10.14778/3377369.3377382},
        url = {https://doi.org/10.14778/3377369.3377382},
        year = {2020}
}

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,000	Permutable Compiled Queries: Dynamically Adapting Compiled Queries without Recompiling	2021	VLDB	6.4069917e-05
8,636	Adaptive Code Generation for Data-Intensive Analytics	2021	VLDB	5.3954959e-05
9,419	Language-Agnostic Integrated Queries in a Managed Polyglot Runtime	2021	VLDB	5.273161e-05

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
23	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00054886415
24	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00054865648
649	Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask	2018	VLDB	0.00015320656
1,379	Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last	2018	VLDB	0.00010970894
2,391	Mison: A Fast JSON Parser for Data Analytics	2017	VLDB	8.6413407e-05
2,414	Filter Before You Parse: Faster Analytics on Raw Data with Sparser	2018	VLDB	8.6078841e-05
2,597	The MemSQL Query Optimizer: A modern optimizer for real-time analytics in a distributed database	2016	VLDB	8.3604418e-05
3,411	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	7.436229e-05
4,481	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	6.6754521e-05
5,048	FAD.js: Fast JSON Data Access Using JIT-based Speculative Optimizations	2017	VLDB	6.3870952e-05
6,181	Just-In-Time Data Virtualization: Lightweight Data Management with ViDa	2015	CIDR	5.9494838e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	6,344	Towards General and Efficient Online Tuning for Spark	2023	VLDB
2	1,190	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
3	3,411	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
4	11,091	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB
5	8,175	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB
6	24	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD
7	8,439	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB
8	3,074	Speculative Distributed CSV Data Parsing for Big Data Analytics	2019	SIGMOD
9	2,594	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
10	8,615	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB