An Empirical Evaluation of Columnar Storage Formats

Summary: A deep empirical comparison of Parquet and ORC under modern hardware, data distributions, and workloads. It derives format-level guidance—default dictionaries, decode-oriented integer codecs, optional compression, finer-grained indexes—and exposes ML/GPU decoding bottlenecks. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: h42f122791246459d
Venue: VLDB
Year: 2024
Pagerank: 7.0064171e-05
Overall Rank: 3,810 | 74.40%
DOI: 10.14778/3626292.3626298
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Xinyu Zeng (Tsinghua University)
2. Yulong Hui (Tsinghua University)
3. Jiahong Shen (Tsinghua University)
4. Andrew Pavlo (Carnegie Mellon University)
5. Wes McKinney (Voltron Data)
6. Huanchen Zhang (Shanghai Qi Zhi Institute; Tsinghua University)

BibTeX Citation

@article{zeng_vldb24,
        title = {{An Empirical Evaluation of Columnar Storage Formats}},
        author = {Zeng, Xinyu and Hui, Yulong and Shen, Jiahong and Pavlo, Andrew and McKinney, Wes and Zhang, Huanchen},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {2},
        pages = {148--161},
        doi = {10.14778/3626292.3626298},
        url = {https://doi.org/10.14778/3626292.3626298},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 21 of 21 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,155	LeCo: Lightweight Compression via Learning Serial Correlations	2024	SIGMOD	6.7730383e-05
4,638	A Deep Dive into Common Open Formats for Analytical DBMSs	2023	VLDB	6.4901967e-05
7,223	Two Birds With One Stone: Designing a Hybrid Cloud Storage Engine for HTAP	2024	VLDB	5.5801574e-05
7,353	Bullion: A Column Store for Machine Learning	2025	CIDR	5.5424978e-05
8,427	Femur: A Flexible Framework for Fast and Secure Querying from Public Key-Value Store	2025	SIGMOD	5.3324907e-05
8,559	F3: The Open-Source Data File Format for the Future	2026	SIGMOD	5.3141911e-05
8,577	The FastLanes File Format	2025	VLDB	5.3107348e-05
9,323	Apache TsFile: An IoT-native Time Series File Format	2024	VLDB	5.1934797e-05
9,690	The HANA Native Query Engine for Lakehouse Systems	2025	VLDB	5.1399285e-05
10,027	Towards Functional Decomposition of Storage Formats	2025	CIDR	5.0917543e-05
10,085	LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics	2025	VLDB	5.0806786e-05
10,311	Data Chunk Compaction in Vectorized Execution	2025	SIGMOD	5.0376863e-05
10,602	RABIT: Efficient Range Queries with Bitmap Indexing	2026	SIGMOD	4.9769913e-05
10,662	Improving LZ4 for Effective Compression and Efficient Query	2026	SIGMOD	4.9769913e-05
10,680	Predictive Translation: High-Performance Buffer Management Without the Trade-Offs	2026	SIGMOD	4.9769913e-05
10,683	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD	4.9769913e-05
10,704	FlatStor: An Efficient Embedded-Index Based Columnar Data Layout for Multimodal Data Workloads	2026	VLDB	4.9769913e-05
10,723	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	4.9769913e-05
10,895	Accelerating String-Heavy Queries with LLM Token Tables	2026	VLDB	4.9769913e-05
11,200	Nested Parquet Is Flat, Why Not Use It? How To Scan Nested Data With On-the-Fly Key Generation and Joins	2025	SIGMOD	4.9769913e-05
11,442	Analyzing Near-Network Hardware Acceleration with Co-Processing on DPUs	2025	VLDB	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 37 of 37 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
12	C-Store: A Column-oriented DBMS	2005	VLDB	0.0006897844
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049821554
48	Weaving Relations for Cache Performance	2001	VLDB	0.00043795812
49	Dremel: Interactive Analysis of Web-Scale Datasets	2010	VLDB	0.0004314366
52	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00041210636
61	Integrating Compression and Execution in Column-Oriented Database Systems	2006	SIGMOD	0.00039236924
114	A Decomposition Storage Model	1985	SIGMOD	0.00031916625
148	Gorilla: A Fast, Scalable, In-Memory Time Series Database	2015	VLDB	0.00029001141
189	Milvus: A Purpose-Built Vector Data Management System	2021	SIGMOD	0.0002585319
233	Amazon Redshift and the Case for Simpler Data Warehouses	2015	SIGMOD	0.00023774958
460	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017850215
873	BitWeaving: Fast Scans for Main Memory Data Processing	2013	SIGMOD	0.00013334067
889	SuRF: Practical Range Query Filtering with Fast Succinct Tries	2018	SIGMOD	0.00013243846
950	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012889569
1,135	Dremel: A Decade of Interactive SQL Analysis at Web Scale	2020	VLDB	0.00011886308
1,269	A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics	2020	SIGMOD	0.00011254742
1,347	Chimp: Efficient Lossless Floating Point Compression for Time Series Databases	2022	VLDB	0.00010945231
1,423	Procella: Unifying serving and analytical data at YouTube	2019	VLDB	0.00010715412
1,515	DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems	2021	VLDB	0.00010418766
1,531	Column Imprints: A Secondary Index Structure	2013	SIGMOD	0.00010344951
1,885	SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures	2014	VLDB	9.4345374e-05
2,317	BtrBlocks: Efficient Columnar Compression for Data Lakes	2023	SIGMOD	8.6492924e-05
2,689	Major Technical Advancements in Apache Hive	2014	SIGMOD	8.1227087e-05
2,839	SNARF: A Learning-Enhanced Range Filter	2022	VLDB	7.9490277e-05
2,954	Column Sketches: A Scan Accelerator for Rapid and Robust Predicate Evaluation	2018	SIGMOD	7.8119682e-05
3,182	The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code	2023	VLDB	7.5589507e-05
3,657	Tile-based Lightweight Integer Compression in GPU	2022	SIGMOD	7.1227107e-05
3,888	Orchestrating Data Placement and Query Execution in Heterogeneous CPU-GPU DBMS	2022	VLDB	6.942437e-05
4,107	Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google	2021	VLDB	6.8020956e-05
4,155	LeCo: Lightweight Compression via Learning Serial Correlations	2024	SIGMOD	6.7730383e-05
5,388	Order-Preserving Key Compression for In-Memory Search Trees	2020	SIGMOD	6.1520776e-05
5,477	Good to the Last Bit: Data-Driven Encoding with CodecDB	2021	SIGMOD	6.1152706e-05
5,929	Self-Organizing Data Containers	2022	CIDR	5.9411211e-05
6,278	Shared Foundations: Modernizing Meta's Data Lakehouse	2023	CIDR	5.8243198e-05
6,395	Selection Pushdown in Column Stores using Bit Manipulation Instructions	2023	SIGMOD	5.7981019e-05
7,035	Wide Table Layout Optimization based on Column Ordering and Duplication	2017	SIGMOD	5.6141148e-05
8,849	Columnar Formats for Schemaless LSM-based Document Stores	2022	VLDB	5.2640252e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	8,215	LST-Bench: Benchmarking Log-Structured Tables in the Cloud	2024	SIGMOD
2	8,577	The FastLanes File Format	2025	VLDB
3	2,317	BtrBlocks: Efficient Columnar Compression for Data Lakes	2023	SIGMOD
4	8,559	F3: The Open-Source Data File Format for the Future	2026	SIGMOD
5	6,769	Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters	2013	VLDB
6	6,300	Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats	2021	VLDB
7	6,395	Selection Pushdown in Column Stores using Bit Manipulation Instructions	2023	SIGMOD
8	10,027	Towards Functional Decomposition of Storage Formats	2025	CIDR
9	2,903	Column-Oriented Storage Techniques for MapReduce	2011	VLDB
10	4,638	A Deep Dive into Common Open Formats for Analytical DBMSs	2023	VLDB