New Query Optimization Techniques in the Spark Engine of Azure Synapse

Summary: Novel exchange placement in Spark-based Azure Synapse reduces data shuffles and enables multi-consumer reuse. Push-downs (aggregates, semi-joins) and peephole tweaks target scale-out stateful ops, achieving 1.8x on TPC-DS vs Spark 3.0.1. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12963
Venue: VLDB
Year: 2022
Pagerank: 4.4957661e-05
Overall Rank: 8,506 | 40.83%
DOI: 10.14778/3503585.3503601

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
5,023	GenRewrite: Query Rewriting via Large Language Models	2026	SIGMOD	5.75363e-05
10,121	TQEx: Tensor-based Query Engine Enhanced by Bridging the Gap	2026	SIGMOD	4.1945683e-05
10,749	Scaling GPU-Accelerated Databases beyond GPU Memory Size	2025	VLDB	4.1945683e-05
11,267	Anser: Adaptive Information Sharing Framework of AnalyticDB	2023	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
51	Including Group-By in Query Optimization	1994	VLDB	0.00067123727
60	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00064439773
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061639801
70	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00059533166
167	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00039180521
218	Aggregate-Query Processing in Data Warehousing Environments	1995	VLDB	0.00033503922
248	Eager Aggregation and Lazy Aggregation	1995	VLDB	0.00030785339
454	An Overview of Query Optimization in Relational Systems	1998	PODS	0.00022734812
940	SharedDB: Killing One Thousand Queries With One Stone	2012	VLDB	0.00015173166
1,922	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	0.00010082599
4,248	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.3247927e-05
5,194	Bitvector-aware Query Optimization for Decision Support Queries	2020	SIGMOD	5.6368209e-05
5,300	Applying Hash Filters To Improving The Execution Of Bushy Trees	1993	VLDB	5.5793265e-05
5,888	Magnet: Push-based Shuffle Service for Large-scale Data Processing	2020	VLDB	5.2873617e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
9,124	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB	4.391961e-05
6,209	AutoExecutor: Predictive Parallelism for Spark SQL Queries	2021	VLDB	5.1565972e-05
11,197	QaaD (Query-as-a-Data): Scalable Execution of Massive Number of Small Queries in Spark	2023	SIGMOD	4.1945683e-05
2,241	Query Optimization in Microsoft SQL Server PDW	2012	SIGMOD	9.2191212e-05
5,014	Dynamically Optimizing Queries over Large Scale Data Platforms	2014	SIGMOD	5.7586174e-05
6,673	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	4.966799e-05
8,197	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB	4.5607121e-05
9,305	Parallelizing Query Optimization on Shared-Nothing Architectures	2016	VLDB	4.3577129e-05
5,297	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5801669e-05
8,617	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4846425e-05