DBScholar

Back to papers

An Empirical Evaluation of Columnar Storage Formats

Summary: A deep empirical comparison of Parquet and ORC under modern hardware, data distributions, and workloads. It derives format-level guidance—default dictionaries, decode-oriented integer codecs, optional compression, finer-grained indexes—and exposes ML/GPU decoding bottlenecks. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID
h42f122791246459d
Venue
VLDB
Year
2024
Pagerank
7.0097354e-05
Overall Rank
3,808 | 74.40%
DOI
10.14778/3626292.3626298

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{zeng_vldb24,
        title = {{An Empirical Evaluation of Columnar Storage Formats}},
        author = {Zeng, Xinyu and Hui, Yulong and Shen, Jiahong and Pavlo, Andrew and McKinney, Wes and Zhang, Huanchen},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {2},
        pages = {148--161},
        doi = {10.14778/3626292.3626298},
        url = {https://doi.org/10.14778/3626292.3626298},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 21 of 21 citing papers.

Rank Citing Paper Year Venue Pagerank
4,153 LeCo: Lightweight Compression via Learning Serial Correlations 2024 SIGMOD 6.776246e-05
4,635 A Deep Dive into Common Open Formats for Analytical DBMSs 2023 VLDB 6.4932705e-05
7,221 Two Birds With One Stone: Designing a Hybrid Cloud Storage Engine for HTAP 2024 VLDB 5.5828002e-05
7,350 Bullion: A Column Store for Machine Learning 2025 CIDR 5.5451228e-05
8,418 Femur: A Flexible Framework for Fast and Secure Querying from Public Key-Value Store 2025 SIGMOD 5.3350162e-05
8,552 F3: The Open-Source Data File Format for the Future 2026 SIGMOD 5.316708e-05
8,569 The FastLanes File Format 2025 VLDB 5.3132501e-05
9,314 Apache TsFile: An IoT-native Time Series File Format 2024 VLDB 5.1959394e-05
9,684 The HANA Native Query Engine for Lakehouse Systems 2025 VLDB 5.1423628e-05
10,022 Towards Functional Decomposition of Storage Formats 2025 CIDR 5.0941658e-05
10,080 LiquidCache: Efficient Pushdown Caching for Cloud-Native Data Analytics 2025 VLDB 5.0830849e-05
10,304 Data Chunk Compaction in Vectorized Execution 2025 SIGMOD 5.0400722e-05
10,591 RABIT: Efficient Range Queries with Bitmap Indexing 2026 SIGMOD 4.9793485e-05
10,651 Improving LZ4 for Effective Compression and Efficient Query 2026 SIGMOD 4.9793485e-05
10,669 Predictive Translation: High-Performance Buffer Management Without the Trade-Offs 2026 SIGMOD 4.9793485e-05
10,672 PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems 2026 SIGMOD 4.9793485e-05
10,694 FlatStor: An Efficient Embedded-Index Based Columnar Data Layout for Multimodal Data Workloads 2026 VLDB 4.9793485e-05
10,713 Robust Predicate Transfer with Dynamic Execution 2026 VLDB 4.9793485e-05
10,886 Accelerating String-Heavy Queries with LLM Token Tables 2026 VLDB 4.9793485e-05
11,191 Nested Parquet Is Flat, Why Not Use It? How To Scan Nested Data With On-the-Fly Key Generation and Joins 2025 SIGMOD 4.9793485e-05
11,436 Analyzing Near-Network Hardware Acceleration with Co-Processing on DPUs 2025 VLDB 4.9793485e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 37 of 37 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
12 C-Store: A Column-oriented DBMS 2005 VLDB 0.00068998927
31 Hive - A Warehousing Solution Over a Map-Reduce Framework 2009 VLDB 0.00049839909
48 Weaving Relations for Cache Performance 2001 VLDB 0.00043805923
49 Dremel: Interactive Analysis of Web-Scale Datasets 2010 VLDB 0.00043160717
52 The Snowflake Elastic Data Warehouse 2016 SIGMOD 0.00041219077
61 Integrating Compression and Execution in Column-Oriented Database Systems 2006 SIGMOD 0.000392237
114 A Decomposition Storage Model 1985 SIGMOD 0.00031928929
148 Gorilla: A Fast, Scalable, In-Memory Time Series Database 2015 VLDB 0.0002900671
194 Milvus: A Purpose-Built Vector Data Management System 2021 SIGMOD 0.00025636725
233 Amazon Redshift and the Case for Simpler Data Warehouses 2015 SIGMOD 0.00023783585
459 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017856221
873 BitWeaving: Fast Scans for Main Memory Data Processing 2013 SIGMOD 0.00013338838
891 SuRF: Practical Range Query Filtering with Fast Succinct Tries 2018 SIGMOD 0.00013245926
950 Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics 2021 CIDR 0.00012895553
1,135 Dremel: A Decade of Interactive SQL Analysis at Web Scale 2020 VLDB 0.00011891907
1,268 A Study of the Fundamental Performance Characteristics of GPUs and CPUs for Database Analytics 2020 SIGMOD 0.0001126007
1,347 Chimp: Efficient Lossless Floating Point Compression for Time Series Databases 2022 VLDB 0.00010950342
1,423 Procella: Unifying serving and analytical data at YouTube 2019 VLDB 0.00010719956
1,515 DSB: A Decision Support Benchmark for Workload-Driven and Traditional Database Systems 2021 VLDB 0.00010417728
1,529 Column Imprints: A Secondary Index Structure 2013 SIGMOD 0.00010348765
1,884 SQL-on-Hadoop: Full Circle Back to Shared-Nothing Database Architectures 2014 VLDB 9.4388097e-05
2,314 BtrBlocks: Efficient Columnar Compression for Data Lakes 2023 SIGMOD 8.6533171e-05
2,689 Major Technical Advancements in Apache Hive 2014 SIGMOD 8.1264718e-05
2,837 SNARF: A Learning-Enhanced Range Filter 2022 VLDB 7.9527715e-05
2,952 Column Sketches: A Scan Accelerator for Rapid and Robust Predicate Evaluation 2018 SIGMOD 7.8153507e-05
3,181 The FastLanes Compression Layout: Decoding >100 Billion Integers per Second with Scalar Code 2023 VLDB 7.5625307e-05
3,655 Tile-based Lightweight Integer Compression in GPU 2022 SIGMOD 7.1260841e-05
3,888 Orchestrating Data Placement and Query Execution in Heterogeneous CPU-GPU DBMS 2022 VLDB 6.945725e-05
4,105 Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google 2021 VLDB 6.8053165e-05
4,153 LeCo: Lightweight Compression via Learning Serial Correlations 2024 SIGMOD 6.776246e-05
5,397 Order-Preserving Key Compression for In-Memory Search Trees 2020 SIGMOD 6.1490013e-05
5,471 Good to the Last Bit: Data-Driven Encoding with CodecDB 2021 SIGMOD 6.1181669e-05
5,929 Self-Organizing Data Containers 2022 CIDR 5.9439349e-05
6,275 Shared Foundations: Modernizing Meta's Data Lakehouse 2023 CIDR 5.8270783e-05
6,392 Selection Pushdown in Column Stores using Bit Manipulation Instructions 2023 SIGMOD 5.800848e-05
7,034 Wide Table Layout Optimization based on Column Ordering and Duplication 2017 SIGMOD 5.6167737e-05
8,840 Columnar Formats for Schemaless LSM-based Document Stores 2022 VLDB 5.2665183e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 8,208 LST-Bench: Benchmarking Log-Structured Tables in the Cloud 2024 SIGMOD
2 8,569 The FastLanes File Format 2025 VLDB
3 2,314 BtrBlocks: Efficient Columnar Compression for Data Lakes 2023 SIGMOD
4 8,552 F3: The Open-Source Data File Format for the Future 2026 SIGMOD
5 6,763 Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters 2013 VLDB
6 6,297 Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats 2021 VLDB
7 6,392 Selection Pushdown in Column Stores using Bit Manipulation Instructions 2023 SIGMOD
8 10,022 Towards Functional Decomposition of Storage Formats 2025 CIDR
9 2,903 Column-Oriented Storage Techniques for MapReduce 2011 VLDB
10 4,635 A Deep Dive into Common Open Formats for Analytical DBMSs 2023 VLDB