Dynamic Speculative Optimizations for SQL Compilation in Apache Spark

Summary: Dynamic speculative optimizations for Spark SQL compilation via runtime profiling and adaptive codegen to reduce data access and deserialization overhead on textual formats. Achieves up to 4.4x speedups on TPC-H with CSV/JSON, illustrating a unique runtime-driven codegen approach for Spark. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12273
Venue: VLDB
Year: 2020
Pagerank: 4.3877539e-05
Overall Rank: 9,122 | 36.61%
DOI: 10.14778/3377369.3377382

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,540	Permutable Compiled Queries: Dynamically Adapting Compiled Queries without Recompiling	2021	VLDB	5.450319e-05
8,623	Adaptive Code Generation for Data-Intensive Analytics	2021	VLDB	4.4786182e-05
9,273	Language-Agnostic Integrated Queries in a Managed Polyglot Runtime	2021	VLDB	4.3615327e-05

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
59	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.0006445664
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061707583
848	Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask	2018	VLDB	0.00015933538
1,865	Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last	2018	VLDB	0.00010271333
2,706	Filter Before You Parse: Faster Analytics on Raw Data with Sparser	2018	VLDB	8.2655235e-05
2,825	Mison: A Fast JSON Parser for Data Analytics	2017	VLDB	8.0575959e-05
3,150	The MemSQL Query Optimizer: A modern optimizer for real-time analytics in a distributed database	2016	VLDB	7.4765759e-05
3,536	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9938207e-05
5,012	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	5.7543101e-05
5,353	FAD.js: Fast JSON Data Access Using JIT-based Speculative Optimizations	2017	VLDB	5.552782e-05
6,403	Just-In-Time Data Virtualization: Lightweight Data Management with ViDa	2015	CIDR	5.0717043e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
6,489	Towards General and Efficient Online Tuning for Spark	2023	VLDB	5.0373773e-05
1,546	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD	0.00011418993
3,536	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9938207e-05
10,872	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB	4.1905499e-05
8,196	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5568952e-05
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061707583
8,502	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB	4.491819e-05
3,432	Speculative Distributed CSV Data Parsing for Big Data Analytics	2019	SIGMOD	7.0974248e-05
3,207	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD	7.3847098e-05
8,585	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4856045e-05