An Empirical Evaluation of Columnar Storage Formats

Summary: A deep empirical comparison of Parquet and ORC under modern hardware, data distributions, and workloads. It derives format-level guidance—default dictionaries, decode-oriented integer codecs, optional compression, finer-grained indexes—and exposes ML/GPU decoding bottlenecks. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID: 13578
Venue: VLDB
Year: 2024
Pagerank: 7.0532293e-05
Overall Rank: 3,877 | 73.41%
DOI: 10.14778/3626292.3626298

Incoming Non-self Citations Over Time

Authors

1. Xinyu Zeng (Tsinghua University)
2. Yulong Hui (Tsinghua University)
3. Jiahong Shen (Tsinghua University)
4. Andrew Pavlo (Carnegie Mellon University)
5. Wes McKinney (Voltron Data)
6. Huanchen Zhang (Shanghai Qi Zhi Institute; Tsinghua University)

BibTeX Citation

@article{zeng_vldb24,
        title = {{An Empirical Evaluation of Columnar Storage Formats}},
        author = {Zeng, Xinyu and Hui, Yulong and Shen, Jiahong and Pavlo, Andrew and McKinney, Wes and Zhang, Huanchen},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {2},
        pages = {148--161},
        doi = {10.14778/3626292.3626298},
        url = {https://doi.org/10.14778/3626292.3626298},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 20 of 20 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
4,065	LeCo: Lightweight Compression via Learning Serial Correlations	2024	SIGMOD	6.930275e-05
5,037	A Deep Dive into Common Open Formats for Analytical DBMSs	2023	VLDB	6.3914026e-05
7,072	Two Birds With One Stone: Designing a Hybrid Cloud Storage Engine for HTAP	2024	VLDB	5.7107965e-05
7,285	Bullion: A Column Store for Machine Learning	2025	CIDR	5.6559604e-05
8,249	Femur: A Flexible Framework for Fast and Secure Querying from Public Key-Value Store	2025	SIGMOD	5.4574671e-05
9,038	The FastLanes File Format	2025	VLDB	5.3270943e-05
9,156	Apache TsFile: An IoT-native Time Series File Format	2024	VLDB	5.3119028e-05
9,277	F3: The Open-Source Data File Format for the Future	2026	SIGMOD	5.2937431e-05
9,836	Towards Functional Decomposition of Storage Formats	2025	CIDR	5.2110542e-05
10,395	RABIT: Efficient Range Queries with Bitmap Indexing	2026	SIGMOD	5.093636e-05
10,464	Improving LZ4 for Effective Compression and Efficient Query	2026	SIGMOD	5.093636e-05
10,482	Predictive Translation: High-Performance Buffer Management Without the Trade-Offs	2026	SIGMOD	5.093636e-05
10,485	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD	5.093636e-05
10,509	FlatStor: An Efficient Embedded-Index Based Columnar Data Layout for Multimodal Data Workloads	2026	VLDB	5.093636e-05
10,529	Robust Predicate Transfer with Dynamic Execution	2026	VLDB	5.093636e-05
10,662	Data Chunk Compaction in Vectorized Execution	2025	SIGMOD	5.093636e-05
10,771	Nested Parquet Is Flat, Why Not Use It? How To Scan Nested Data With On-the-Fly Key Generation and Joins	2025	SIGMOD	5.093636e-05
10,997	The HANA Native Query Engine for Lakehouse Systems	2025	VLDB	5.093636e-05
11,078	LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics	2025	VLDB	5.093636e-05
11,080	Analyzing Near-Network Hardware Acceleration with Co-Processing on DPUs	2025	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 37 of 37 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
12	C-Store: A Column-oriented DBMS	2005	VLDB	0.00069513174
32	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00050111008
49	Weaving Relations for Cache Performance	2001	VLDB	0.00043781096
51	Dremel: Interactive Analysis of Web-Scale Datasets	2010	VLDB	0.0004291425
60	Integrating Compression and Execution in Column-Oriented Database Systems	2006	SIGMOD	0.0003955489
66	The Snowflake Elastic Data Warehouse	2016	SIGMOD	0.00038561587
115	A Decomposition Storage Model	1985	SIGMOD	0.00032338948
148	Gorilla: A Fast, Scalable, In-Memory Time Series Database	2015	VLDB	0.00029250767
237	Amazon Redshift and the Case for Simpler Data Warehouses	2015	SIGMOD	0.0002369895
286	Milvus: A Purpose-Built Vector Data Management System	2021	SIGMOD	0.00022357911
520	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017136828
870	BitWeaving: Fast Scans for Main Memory Data Processing	2013	SIGMOD	0.0001350293
880	SuRF: Practical Range Query Filtering with Fast Succinct Tries	2018	SIGMOD	0.00013432693
1,138	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012023643
1,364	Chimp: Efficient Lossless Floating Point Compression for Time Series Databases	2022	VLDB	0.00011020069
1,453	Dremel: A Decade of Interactive SQL Analysis at Web Scale	2020	VLDB	0.00010742227
1,466	A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics	2020	SIGMOD	0.0001068941
1,494	Procella: Unifying serving and analytical data at YouTube	2019	VLDB	0.00010577585
1,507	Column Imprints: A Secondary Index Structure	2013	SIGMOD	0.0001054177
1,832	DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems	2021	VLDB	9.6607418e-05
1,889	SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures	2014	VLDB	9.5335988e-05
2,706	Major Technical Advancements in Apache Hive	2014	SIGMOD	8.2287564e-05
2,788	BtrBlocks: Efficient Columnar Compression for Data Lakes	2023	SIGMOD	8.1205155e-05
2,919	SNARF: A Learning-Enhanced Range Filter	2022	VLDB	7.9628657e-05
2,937	Column Sketches: A Scan Accelerator for Rapid and Robust Predicate Evaluation	2018	SIGMOD	7.9435581e-05
3,592	The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code	2023	VLDB	7.2774889e-05
4,017	Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google	2021	VLDB	6.9525581e-05
4,065	LeCo: Lightweight Compression via Learning Serial Correlations	2024	SIGMOD	6.930275e-05
4,144	Tile-based Lightweight Integer Compression in GPU	2022	SIGMOD	6.8744592e-05
4,243	Orchestrating Data Placement and Query Execution in Heterogeneous CPU-GPU DBMS	2022	VLDB	6.8073248e-05
5,285	Order-Preserving Key Compression for In-Memory Search Trees	2020	SIGMOD	6.2821588e-05
5,622	Good to the Last Bit: Data-Driven Encoding with CodecDB	2021	SIGMOD	6.1461066e-05
6,076	Self-Organizing Data Containers	2022	CIDR	5.9852895e-05
6,407	Shared Foundations: Modernizing Meta's Data Lakehouse	2023	CIDR	5.8850639e-05
6,896	Wide Table Layout Optimization based on Column Ordering and Duplication	2017	SIGMOD	5.7440611e-05
7,139	Selection Pushdown in Column Stores using Bit Manipulation Instructions	2023	SIGMOD	5.6932481e-05
8,682	Columnar Formats for Schemaless LSM-based Document Stores	2022	VLDB	5.3857524e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	2,706	Major Technical Advancements in Apache Hive	2014	SIGMOD
2	9,038	The FastLanes File Format	2025	VLDB
3	2,788	BtrBlocks: Efficient Columnar Compression for Data Lakes	2023	SIGMOD
4	9,277	F3: The Open-Source Data File Format for the Future	2026	SIGMOD
5	6,640	Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters	2013	VLDB
6	6,264	Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats	2021	VLDB
7	7,139	Selection Pushdown in Column Stores using Bit Manipulation Instructions	2023	SIGMOD
8	9,836	Towards Functional Decomposition of Storage Formats	2025	CIDR
9	2,849	Column-Oriented Storage Techniques for MapReduce	2011	VLDB
10	5,037	A Deep Dive into Common Open Formats for Analytical DBMSs	2023	VLDB