Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE
Summary: Hyper Dimension Shuffle introduces a recursive, divide-and-conquer shuffle for petabyte-scale data in SCOPE. Recursive partitioning with intermediate aggregations yields quasilinear shuffling complexity and tight fan-out/fan-in guarantees, avoiding prior quadratic blowups. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Shi Qiao (Microsoft)
- 2. Adrian Nicoara (Microsoft)
- 3. Jin Sun (Microsoft)
- 4. Marc Friedman (Microsoft)
- 5. Hiren Patel (Microsoft)
- 6. Jaliya Ekanayake (Microsoft)
BibTeX Citation
@article{qiao_vldb19,
title = {{Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE}},
author = {Qiao, Shi and Nicoara, Adrian and Sun, Jin and Friedman, Marc and Patel, Hiren and Ekanayake, Jaliya},
journal = {PVLDB},
series = {{VLDB} '19},
volume = {12},
number = {10},
pages = {1113--1125},
doi = {10.14778/3339490.3339495},
url = {https://doi.org/10.14778/3339490.3339495},
year = {2019}
}Incoming Citations (Sorted by Pagerank)
Showing 9 of 9 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,134 | Dremel: A Decade of Interactive SQL Analysis at Web Scale | 2020 | VLDB | 0.00011863043 |
| 5,529 | Magnet: Push-based Shuffle Service for Large-scale Data Processing | 2020 | VLDB | 6.0789442e-05 |
| 6,260 | The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward | 2021 | VLDB | 5.815273e-05 |
| 6,294 | Incorporating Super-Operators in Big-Data Query Optimizers | 2020 | VLDB | 5.8043081e-05 |
| 7,108 | KEA: Tuning an Exabyte-Scale Data Infrastructure | 2021 | SIGMOD | 5.5817869e-05 |
| 7,944 | Runtime Variation in Big Data Analytics | 2023 | SIGMOD | 5.4051579e-05 |
| 8,539 | New Query Optimization Techniques in the Spark Engine of Azure Synapse | 2022 | VLDB | 5.3048225e-05 |
| 9,117 | Hyperspace: The Indexing Subsystem of Azure Synapse | 2021 | VLDB | 5.208561e-05 |
| 10,985 | Reaching the Pinnacle of TPC-DS: Co-design of Architecture, Executor, and Storage in TDSQL | 2026 | VLDB | 4.9605343e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 9 of 9 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 30 | SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets | 2008 | VLDB | 0.00050364918 |
| 120 | HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads | 2009 | VLDB | 0.00031017868 |
| 677 | Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) | 2010 | VLDB | 0.00014835154 |
| 1,205 | NUMA-aware algorithms: the case of data shuffling | 2013 | CIDR | 0.00011512846 |
| 1,254 | Blink and It's Done: Interactive Queries on Very Large Data | 2012 | VLDB | 0.00011314204 |
| 2,244 | Shark: Fast Data Analysis Using Coarse-grained Distributed Memory | 2012 | SIGMOD | 8.7648296e-05 |
| 3,466 | Scaling Spark in the Real World: Performance and Usability | 2015 | VLDB | 7.2609893e-05 |
| 3,550 | Computation Reuse in Analytics Job Service at Microsoft | 2018 | SIGMOD | 7.1895215e-05 |
| 7,857 | Quill: Efficient, Transferable, and Rich Analytics at Scale | 2016 | VLDB | 5.4230146e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 12,531 | A Partitioning Framework for Aggressive Data Skipping | 2014 | VLDB |
| 2 | 1,215 | Bucket Spreading Parallel Hash: A New, Robust, Parallel Hash Join Method for Data Skew in the Super Database Computer (SDC) | 1990 | VLDB |
| 3 | 1,025 | High-Speed Query Processing over High-Speed Networks | 2016 | VLDB |
| 4 | 5,288 | Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner | 2017 | VLDB |
| 5 | 3,638 | Advanced Join Strategies for Large-Scale Distributed Computation | 2014 | VLDB |
| 6 | 12,380 | An Efficient MapReduce Cube Algorithm for Varied Data Distributions | 2016 | SIGMOD |
| 7 | 4,550 | AdaptDB: Adaptive Partitioning for Distributed Joins | 2017 | VLDB |
| 8 | 4,842 | Continuous Cloud-Scale Query Optimization and Processing | 2013 | VLDB |
| 9 | 6,294 | Incorporating Super-Operators in Big-Data Query Optimizers | 2020 | VLDB |
| 10 | 3,453 | Advanced Partitioning Techniques for Massively Distributed Computation | 2012 | SIGMOD |