DBScholar

Back to papers

An Empirical Evaluation of Columnar Storage Formats

Summary: A deep empirical comparison of Parquet and ORC under modern hardware, data distributions, and workloads. It derives format-level guidance—default dictionaries, decode-oriented integer codecs, optional compression, finer-grained indexes—and exposes ML/GPU decoding bottlenecks. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID
h42f122791246459d
Venue
VLDB
Year
2024
Pagerank
6.9890843e-05
Overall Rank
3,809 | 74.47%
DOI
10.14778/3626292.3626298
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{zeng_vldb24,
        title = {{An Empirical Evaluation of Columnar Storage Formats}},
        author = {Zeng, Xinyu and Hui, Yulong and Shen, Jiahong and Pavlo, Andrew and McKinney, Wes and Zhang, Huanchen},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {2},
        pages = {148--161},
        doi = {10.14778/3626292.3626298},
        url = {https://doi.org/10.14778/3626292.3626298},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 22 of 22 citing papers.

Rank Citing Paper Year Venue Pagerank
4,169 LeCo: Lightweight Compression via Learning Serial Correlations 2024 SIGMOD 6.7526687e-05
4,373 A Deep Dive into Common Open Formats for Analytical DBMSs 2023 VLDB 6.6191786e-05
6,994 Two Birds With One Stone: Designing a Hybrid Cloud Storage Engine for HTAP 2024 VLDB 5.6094434e-05
7,369 Bullion: A Column Store for Machine Learning 2025 CIDR 5.5261972e-05
7,771 The FastLanes File Format 2025 VLDB 5.4390723e-05
8,443 Femur: A Flexible Framework for Fast and Secure Querying from Public Key-Value Store 2025 SIGMOD 5.3148581e-05
8,578 F3: The Open-Source Data File Format for the Future 2026 SIGMOD 5.2966191e-05
9,349 Apache TsFile: An IoT-native Time Series File Format 2024 VLDB 5.1763069e-05
9,719 The HANA Native Query Engine for Lakehouse Systems 2025 VLDB 5.1229327e-05
10,065 Towards Functional Decomposition of Storage Formats 2025 CIDR 5.0749178e-05
10,119 LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics 2025 VLDB 5.0638787e-05
10,339 Data Chunk Compaction in Vectorized Execution 2025 SIGMOD 5.0210286e-05
10,399 G-CasDec: General Cascaded Decompression on GPUs 2027 SIGMOD 4.9605343e-05
10,664 RABIT: Efficient Range Queries with Bitmap Indexing 2026 SIGMOD 4.9605343e-05
10,723 Improving LZ4 for Effective Compression and Efficient Query 2026 SIGMOD 4.9605343e-05
10,741 Predictive Translation: High-Performance Buffer Management Without the Trade-Offs 2026 SIGMOD 4.9605343e-05
10,744 PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems 2026 SIGMOD 4.9605343e-05
10,765 FlatStor: An Efficient Embedded-Index Based Columnar Data Layout for Multimodal Data Workloads 2026 VLDB 4.9605343e-05
10,784 Robust Predicate Transfer with Dynamic Execution 2026 VLDB 4.9605343e-05
10,956 Accelerating String-Heavy Queries with LLM Token Tables 2026 VLDB 4.9605343e-05
11,259 Nested Parquet Is Flat, Why Not Use It? How To Scan Nested Data With On-the-Fly Key Generation and Joins 2025 SIGMOD 4.9605343e-05
11,494 Analyzing Near-Network Hardware Acceleration with Co-Processing on DPUs 2025 VLDB 4.9605343e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 37 of 37 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
12 C-Store: A Column-oriented DBMS 2005 VLDB 0.00068985264
31 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00049707539
48 Weaving Relations for Cache Performance 2001 VLDB 0.00043770511
49 Dremel: Interactive Analysis of Web-Scale Datasets 2010 VLDB 0.00043085722
52 The Snowflake Elastic Data Warehouse 2016 SIGMOD 0.00041439488
61 Integrating Compression and Execution in Column-Oriented Database Systems 2006 SIGMOD 0.00039323387
114 A Decomposition Storage Model 1985 SIGMOD 0.00031854546
147 Gorilla: A Fast, Scalable, In-Memory Time Series Database 2015 VLDB 0.00028941363
181 Milvus: A Purpose-Built Vector Data Management System 2021 SIGMOD 0.00026425782
226 Amazon Redshift and the Case for Simpler Data Warehouses 2015 SIGMOD 0.0002389883
462 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017825062
871 BitWeaving: Fast Scans for Main Memory Data Processing 2013 SIGMOD 0.00013313281
892 SuRF: Practical Range Query Filtering with Fast Succinct Tries 2018 SIGMOD 0.00013207671
955 Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics 2021 CIDR 0.00012856295
1,134 Dremel: A Decade of Interactive SQL Analysis at Web Scale 2020 VLDB 0.00011863043
1,228 A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics 2020 SIGMOD 0.00011425748
1,351 Chimp: Efficient Lossless Floating Point Compression for Time Series Databases 2022 VLDB 0.00010921539
1,387 Procella: Unifying serving and analytical data at YouTube 2019 VLDB 0.0001080904
1,513 DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems 2021 VLDB 0.00010394557
1,532 Column Imprints: A Secondary Index Structure 2013 SIGMOD 0.00010312002
1,889 SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures 2014 VLDB 9.4145058e-05
2,239 BtrBlocks: Efficient Columnar Compression for Data Lakes 2023 SIGMOD 8.7753778e-05
2,700 Major Technical Advancements in Apache Hive 2014 SIGMOD 8.096888e-05
2,844 SNARF: A Learning-Enhanced Range Filter 2022 VLDB 7.9291659e-05
2,965 Column Sketches: A Scan Accelerator for Rapid and Robust Predicate Evaluation 2018 SIGMOD 7.7894987e-05
3,063 The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code 2023 VLDB 7.6806166e-05
3,471 Tile-based Lightweight Integer Compression in GPU 2022 SIGMOD 7.258594e-05
3,828 Orchestrating Data Placement and Query Execution in Heterogeneous CPU-GPU DBMS 2022 VLDB 6.9779559e-05
4,126 Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google 2021 VLDB 6.7797675e-05
4,169 LeCo: Lightweight Compression via Learning Serial Correlations 2024 SIGMOD 6.7526687e-05
5,398 Order-Preserving Key Compression for In-Memory Search Trees 2020 SIGMOD 6.1338952e-05
5,489 Good to the Last Bit: Data-Driven Encoding with CodecDB 2021 SIGMOD 6.0972774e-05
5,935 Self-Organizing Data Containers 2022 CIDR 5.9251217e-05
6,290 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 5.8051881e-05
6,400 Selection Pushdown in Column Stores using Bit Manipulation Instructions 2023 SIGMOD 5.7811299e-05
7,054 Wide Table Layout Optimization based on Column Ordering and Duplication 2017 SIGMOD 5.5956321e-05
8,868 Columnar Formats for Schemaless LSM-based Document Stores 2022 VLDB 5.2467001e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 2,700 Major Technical Advancements in Apache Hive 2014 SIGMOD
2 7,771 The FastLanes File Format 2025 VLDB
3 2,239 BtrBlocks: Efficient Columnar Compression for Data Lakes 2023 SIGMOD
4 8,578 F3: The Open-Source Data File Format for the Future 2026 SIGMOD
5 6,785 Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters 2013 VLDB
6 6,295 Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats 2021 VLDB
7 6,400 Selection Pushdown in Column Stores using Bit Manipulation Instructions 2023 SIGMOD
8 10,065 Towards Functional Decomposition of Storage Formats 2025 CIDR
9 2,918 Column-Oriented Storage Techniques for MapReduce 2011 VLDB
10 4,373 A Deep Dive into Common Open Formats for Analytical DBMSs 2023 VLDB