Data Chunk Compaction in Vectorized Execution

Summary: Addresses data-chunk fragmentation in vectorized analytics caused by hash-join processing. Proposes a runtime-learning compaction threshold and hash-join-specific logical compaction to minimize movement, delivering up to 63% speedups in DuckDB on TPC-H/DS. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hedb995c67bf591e7
Venue: SIGMOD
Year: 2025
Pagerank: 5.0376863e-05
Overall Rank: 10,311 | 30.70%
DOI: 10.1145/3709676
PDF: Download (CC BY 4.0)

Incoming Non-self Citations Over Time

Authors

1. Yiming Qiao (Tsinghua University)
2. Huanchen Zhang (Shanghai Qi Zhi Institute; Tsinghua University)

BibTeX Citation

@inproceedings{qiao_sigmod25,
        title = {{Data Chunk Compaction in Vectorized Execution}},
        author = {Qiao, Yiming and Zhang, Huanchen},
        series = {{SIGMOD} '25},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3709676},
        url = {https://dl.acm.org/doi/10.1145/3709676},
        year = {2025}
}

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,945	Debunking the Myth of Join Ordering: Toward Robust SQL Analytics	2025	SIGMOD	6.3418058e-05
10,723	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	4.9769913e-05
10,941	IMLane: Composable Framework for Efficient AI Function Execution in Database Engine	2026	VLDB	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 31 of 31 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
14	MonetDB/X100: Hyper-Pipelining Query Execution	2005	CIDR	0.00064013679
15	How Good Are Query Optimizers, Really?	2016	VLDB	0.00061067652
52	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00041210636
61	Integrating Compression and Execution in Column-Oriented Database Systems	2006	SIGMOD	0.00039236924
71	DuckDB: an Embeddable Analytical Database	2019	SIGMOD	0.00037724477
76	Cache Conscious Algorithms for Relational Query Processing	1994	VLDB	0.00036891569
163	DB2 with BLU Acceleration: So Much More than Just a Column Store	2013	VLDB	0.00027480091
185	CockroachDB: The Resilient Geo-Distributed SQL Database	2020	SIGMOD	0.00026060465
210	Sort vs. Hash Revisited: Fast Join Implementation on Modern Multi-Core CPUs	2009	VLDB	0.00024844328
215	Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age	2014	SIGMOD	0.00024589307
605	Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask	2018	VLDB	0.00015640305
627	Rethinking SIMD Vectorization for In-Memory Databases	2015	SIGMOD	0.00015454197
907	Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation	2016	SIGMOD	0.00013157412
1,135	Dremel: A Decade of Interactive SQL Analysis at Web Scale	2020	VLDB	0.00011886308
1,267	An Experimental Comparison of Thirteen Relational Equi-Joins in Main Memory	2016	SIGMOD	0.00011265987
1,341	Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last	2018	VLDB	0.00010963427
1,423	Procella: Unifying serving and analytical data at YouTube	2019	VLDB	0.00010715412
1,424	Velox: Meta's Unified Execution Engine	2022	VLDB	0.00010714183
1,487	Photon: A Fast Query Engine for Lakehouse Systems	2022	SIGMOD	0.00010516813
2,802	Distributed Join Algorithms on Thousands of Cores	2017	VLDB	7.9866934e-05
2,818	To Partition, or Not to Partition, That is the Join Question in a Real System	2021	SIGMOD	7.9703078e-05
3,003	ClickHouse - Lightning Fast Analytics for Everyone	2024	VLDB	7.7631056e-05
3,102	Micro Adaptivity in Vectorwise	2013	SIGMOD	7.6469535e-05
3,810	An Empirical Evaluation of Columnar Storage Formats	2024	VLDB	7.0064171e-05
4,074	Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine	2024	SIGMOD	6.8175757e-05
5,545	Charting the Design Space of Query Execution using VOILA	2021	VLDB	6.0860849e-05
7,387	From X100 to Vectorwise: Opportunities, Challenges and Things Most Researchers do not Think About	2012	SIGMOD	5.5359243e-05
7,808	Analyzing Vectorized Hash Tables Across CPU Architectures	2023	VLDB	5.448023e-05
8,170	Efficient Query Re-optimization with Judicious Subquery Selections	2023	SIGMOD	5.3827384e-05
10,196	Towards Unifying Query Interpretation and Compilation	2023	CIDR	5.0609527e-05
11,585	Blitzcrank: Fast Semantic Compression for In-memory Online Transaction Processing	2024	VLDB	4.9769913e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	898	Query Optimization In Compressed Database Systems	2001	SIGMOD
2	362	Design and Evaluation of Main Memory Hash Join Algorithms for Multi-core CPUs	2011	SIGMOD
3	10,778	One Join Order Does Not Fit All: Reducing Intermediate Results with Per-Split Query Plans	2026	VLDB
4	907	Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation	2016	SIGMOD
5	8,570	Robust and Budget-Constrained Encoding Configurations for In-Memory Database Systems	2022	VLDB
6	1,069	Fast Computation of Sparse Datacubes	1997	VLDB
7	5,427	Joins on Encoded and Partitioned Data	2014	VLDB
8	5,492	Compressed Representations of Conjunctive Query Results	2018	PODS
9	7,768	Adaptive Factorization Using Linear-Chained Hash Tables	2025	CIDR
10	9,574	Saving Private Hash Join	2025	VLDB