Adaptive and Robust Query Execution for Lakehouses at Scale

Summary: AQE for lakehouses: use pipeline breakers to collect runtime statistics and reoptimize plans, mitigating missing/incorrect table/column stats and bad cardinality/UDF estimates. Up to 25× TPC‑DS speedup; deployed at Databricks for exabyte‑scale workloads to reduce data movement, spills, and memory pressure. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13596
Venue: VLDB
Year: 2024
Pagerank: 4.8477825e-05
Overall Rank: 7,059 | 50.90%
DOI: 10.14778/3685800.3685818

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 9 of 9 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
6,895	Decentralized Actor Scheduling and Reference-based Storage in Xorbits: a Native Scalable Data Science Engine	2025	VLDB	4.8925595e-05
9,093	Databricks Lakeguard: Supporting Fine-grained Access Control and Multi-user Capabilities for Apache Spark Workloads	2025	SIGMOD	4.398149e-05
9,747	Still Asking: How Good Are Query Optimizers, Really?	2025	VLDB	4.2897489e-05
9,763	The UDFBench Benchmark for General-purpose UDF Queries	2025	VLDB	4.2856106e-05
10,241	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	4.1945683e-05
10,767	The HANA Native Query Engine for Lakehouse Systems	2025	VLDB	4.1945683e-05
10,772	veDB-HTAP: a Highly Integrated, Efficient and Adaptive HTAP System	2025	VLDB	4.1945683e-05
10,859	Graph Transformers for Query Plan Representation: Potentials and Challenges	2025	VLDB	4.1945683e-05
13,096	Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries	2025	SIGMOD	-

Outgoing Citations (Sorted by Pagerank)

Showing 24 of 24 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
1	Access Path Selection in a Relational Database Management System	1979	SIGMOD	0.0040449103
66	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00061639801
115	Eddies: Continuously Adaptive Query Processing	2000	SIGMOD	0.00046221215
167	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00039180521
182	LEO - DB2's LEarning Optimizer	2001	VLDB	0.00036962631
220	Efficient Mid-Query Re-Optimization of Sub-Optimal Query Execution Plans	1998	SIGMOD	0.00033194808
310	The Vertica Analytic Database: C-Store 7 Years Later	2012	VLDB	0.00028132402
426	Amazon Redshift and the Case for Simpler Data Warehouses	2015	SIGMOD	0.00023594359
476	Impala: A Modern, Open-Source SQL Engine for Hadoop	2015	CIDR	0.00022226941
508	Dynamic Query Evaluation Plans	1989	SIGMOD	0.00021463742
520	An Overview of The System Software of A Parallel Relational Database Machine GRACE	1986	VLDB	0.00021152636
542	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00020595648
650	Robust Query Processing through Progressive Optimization	2004	SIGMOD	0.00018659177
746	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017326979
1,377	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012296941
1,574	Approximate Query Processing: No Silver Bullet	2017	SIGMOD	0.00011287495
1,797	Effective Use of Block-Level Sampling in Statistics Estimation	2004	SIGMOD	0.00010523169
1,915	Handling Data Skew in Parallel Joins in Shared-Nothing Systems	2008	SIGMOD	0.00010104123
2,473	Photon: A Fast Query Engine for Lakehouse Systems	2022	SIGMOD	8.7237281e-05
2,504	Enhanced Subquery Optimizations in Oracle	2009	VLDB	8.6351917e-05
2,940	Operating System Extensions for the Teradata Parallel VLDB	2001	VLDB	7.8520709e-05
3,355	F1 Query: Declarative Querying at Scale	2018	VLDB	7.1829142e-05
5,531	Presto: A Decade of SQL Analytics at Meta	2023	SIGMOD	5.4549499e-05
6,390	Proactive Re-optimization with Rio	2005	SIGMOD	5.0842083e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
5,318	Analyzing and Comparing Lakehouse Storage Systems	2023	CIDR	5.5715872e-05
13,359	Robust Data Transformations	2015	CIDR	-
5,297	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5801669e-05
8,582	Towards Query Optimizer as a Service (QOaaS) in a Unified LakeHouse Ecosystem: Can One QO Rule Them All?	2025	CIDR	4.492033e-05
2,473	Photon: A Fast Query Engine for Lakehouse Systems	2022	SIGMOD	8.7237281e-05
6,402	BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse	2024	SIGMOD	5.079818e-05
8,617	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB	4.4846425e-05
1,429	A Scalable, Predictable Join Operator for Highly Concurrent Data Warehouses	2009	VLDB	0.00012033518
10,248	Active Data Lakes: Regaining Physical Data Independence Without Losing Interoperability	2026	VLDB	4.1945683e-05
7,907	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB	4.6205839e-05