Dynamic Speculative Optimizations for SQL Compilation in Apache Spark

Summary: Dynamic speculative optimizations for Spark SQL compilation via runtime profiling and adaptive codegen to reduce data access and deserialization overhead on textual formats. Achieves up to 4.4x speedups on TPC-H with CSV/JSON, illustrating a unique runtime-driven codegen approach for Spark. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12272
Venue: VLDB
Year: 2020
Pagerank: 4.391961e-05
Overall Rank: 9,124 | 36.53%
DOI: 10.14778/3377369.3377382

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,530	Permutable Compiled Queries: Dynamically Adapting Compiled Queries without Recompiling	2021	VLDB	5.4554282e-05
8,626	Adaptive Code Generation for Data-Intensive Analytics	2021	VLDB	4.4829152e-05
9,268	Language-Agnostic Integrated Queries in a Managed Polyglot Runtime	2021	VLDB	4.3657168e-05

Outgoing Citations (Sorted by Pagerank)

Showing 11 of 11 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
60	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00064439773
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061639801
853	Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask	2018	VLDB	0.00015940507
1,864	Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last	2018	VLDB	0.00010280966
2,700	Filter Before You Parse: Faster Analytics on Raw Data with Sparser	2018	VLDB	8.2728509e-05
2,819	Mison: A Fast JSON Parser for Data Analytics	2017	VLDB	8.0651326e-05
3,154	The MemSQL Query Optimizer: A modern optimizer for real-time analytics in a distributed database	2016	VLDB	7.4686089e-05
3,535	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9992495e-05
5,014	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	5.7586174e-05
5,343	FAD.js: Fast JSON Data Access Using JIT-based Speculative Optimizations	2017	VLDB	5.5581129e-05
6,407	Just-In-Time Data Virtualization: Lightweight Data Management with ViDa	2015	CIDR	5.076547e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
6,871	Towards General and Efficient Online Tuning for Spark	2023	VLDB	4.8997004e-05
1,548	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD	0.00011431383
3,535	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9992495e-05
10,868	LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison	2025	VLDB	4.1945683e-05
8,197	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5607121e-05
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061639801
8,506	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB	4.4957661e-05
3,437	Speculative Distributed CSV Data Parsing for Big Data Analytics	2019	SIGMOD	7.0942161e-05
3,200	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD	7.3912411e-05
8,617	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4846425e-05