DBScholar

Back to papers

Partition, Don’t Sort! Compression Boosters for Cloud Data Ingestion Pipelines

Summary: Rather than expensive global sorting, cluster similarly structured nested Dremel-encoded records at ingestion to create compressible partitions. A decision-tree–inspired clustering is up to 17.44× faster than partition-then-sort and yields up to 2× compression, while per-bucket sorting matches increasing-cardinality compression at lower ingestion cost. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID
h9c71d43be130b0c1
Venue
VLDB
Year
2024
Pagerank
4.9605343e-05
Overall Rank
11,654 | 21.89%
DOI
10.14778/3681954.3682013
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

BibTeX Citation

@article{hansert_vldb24,
        title = {{Partition, Don’t Sort! Compression Boosters for Cloud Data Ingestion Pipelines}},
        author = {Hansert, Patrick and Michel, Sebastian},
        journal = {PVLDB},
        series = {{VLDB} '24},
        volume = {17},
        number = {11},
        pages = {3456--3469},
        doi = {10.14778/3681954.3682013},
        url = {https://doi.org/10.14778/3681954.3682013},
        year = {2024}
}

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank Citing Paper Year Venue Pagerank
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 23 of 23 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
20 Similarity Search in High Dimensions via Hashing 1999 VLDB 0.00057610161
49 Dremel: Interactive Analysis of Web-Scale Datasets 2010 VLDB 0.00043085722
61 Integrating Compression and Execution in Column-Oriented Database Systems 2006 SIGMOD 0.00039323387
253 Database Cracking 2007 CIDR 0.00023037921
363 Linear Clustering of Objects with Multiple Attributes 1990 SIGMOD 0.00019931086
462 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017825062
473 The Making of TPC-DS 2006 VLDB 0.00017656775
980 Sybase IQ Multiplex – Designed For Analytics 2004 VLDB 0.0001268915
1,134 Dremel: A Decade of Interactive SQL Analysis at Web Scale 2020 VLDB 0.00011863043
2,772 Instance-Optimized Data Layouts for Cloud Analytics Workloads 2021 SIGMOD 8.0148928e-05
3,094 Skipping-oriented Partitioning for Columnar Layouts 2017 VLDB 7.637515e-05
3,744 NET-FLi: On-the-fly Compression, Archiving and Indexing of Streaming Network Traffic 2010 VLDB 7.0372503e-05
3,927 JSON Tiles: Fast Analytics on Semi-Structured Data 2021 SIGMOD 6.9025015e-05
5,856 Pando: Enhanced Data Skipping with Logical Data Partitioning 2023 VLDB 5.9514226e-05
6,041 Automated Multidimensional Data Layouts in Amazon Redshift 2024 SIGMOD 5.8894534e-05
6,101 Column Partition and Permutation for Run Length Encoding in Columnar Databases 2020 SIGMOD 5.8695901e-05
6,429 Proteus: Autonomous Adaptive Storage for Mixed Workloads 2022 SIGMOD 5.7691021e-05
6,532 Exploiting Common Patterns for Tree-Structured Data 2017 SIGMOD 5.7362407e-05
6,785 Understanding Insights into the Basic Structure and Essential Issues of Table Placement Methods in Clusters 2013 VLDB 5.6659903e-05
6,977 Jigsaw: A Data Storage and Query Processing Engine for Irregular Table Partitioning 2021 SIGMOD 5.6135617e-05
7,046 Rearranging Data to Maximize the Efficiency of Compression 1986 PODS 5.5975398e-05
7,054 Wide Table Layout Optimization based on Column Ordering and Duplication 2017 SIGMOD 5.5956321e-05
7,484 Reducing Ambiguity in Json Schema Discovery 2021 SIGMOD 5.5005016e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 10,339 Data Chunk Compaction in Vectorized Execution 2025 SIGMOD
2 8,439 Workload-Aware Incremental Reclustering in Cloud Data Warehouses 2026 SIGMOD
3 5,442 Joins on Encoded and Partitioned Data 2014 VLDB
4 9,874 High-Ratio Compression for Machine-Generated Data 2023 SIGMOD
5 2,481 Learning a Partitioning Advisor for Cloud Databases 2020 SIGMOD
6 8,589 Robust and Budget-Constrained Encoding Configurations for In-Memory Database Systems 2022 VLDB
7 6,737 CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases 2022 SIGMOD
8 2,239 BtrBlocks: Efficient Columnar Compression for Data Lakes 2023 SIGMOD
9 6,101 Column Partition and Permutation for Run Length Encoding in Columnar Databases 2020 SIGMOD
10 10,065 Towards Functional Decomposition of Storage Formats 2025 CIDR