Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE

Summary: Hyper Dimension Shuffle introduces a recursive, divide-and-conquer shuffle for petabyte-scale data in SCOPE. Recursive partitioning with intermediate aggregations yields quasilinear shuffling complexity and tight fan-out/fan-in guarantees, avoiding prior quadratic blowups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11811
Venue: VLDB
Year: 2019
Pagerank: 6.3382156e-05
Overall Rank: 4,226 | 70.64%
DOI: 10.14778/3339490.3339495

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,060	Dremel: A Decade of Interactive SQL Analysis at Web Scale	2020	VLDB	9.6585115e-05
5,587	Magnet: Push-based Shuffle Service for Large-scale Data Processing	2020	VLDB	5.4193445e-05
6,278	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.1241654e-05
6,671	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	4.9625353e-05
7,099	KEA: Tuning an Exabyte-Scale Data Infrastructure	2021	SIGMOD	4.8263529e-05
7,778	Runtime Variation in Big Data Analytics	2023	SIGMOD	4.6491879e-05
8,502	New Query Optimization Techniques in the Spark Engine of Azure Synapse	2022	VLDB	4.491819e-05
8,754	Hyperspace: The Indexing Subsystem of Azure Synapse	2021	VLDB	4.4520434e-05

Outgoing Citations (Sorted by Pagerank)

Showing 9 of 9 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
22	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00084679526
158	HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads	2009	VLDB	0.00040401371
789	Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing)	2010	VLDB	0.00016602215
1,151	Blink and It's Done: Interactive Queries on Very Large Data	2012	VLDB	0.00013634671
1,540	NUMA-aware algorithms: the case of data shuffling	2013	CIDR	0.00011451745
2,487	Shark: Fast Data Analysis Using Coarse-grained Distributed Memory	2012	SIGMOD	8.6609871e-05
3,536	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	6.9938207e-05
4,171	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	6.3800823e-05
7,600	Quill: Efficient, Transferable, and Rich Analytics at Scale	2016	VLDB	4.697054e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
12,001	A Partitioning Framework for Aggressive Data Skipping	2014	VLDB	4.1905499e-05
1,219	Bucket Spreading Parallel Hash: A New, Robust, Parallel Hash Join Method for Data Skew in the Super Database Computer (SDC)	1990	VLDB	0.00013215689
1,397	High-Speed Query Processing over High-Speed Networks	2016	VLDB	0.00012208385
4,899	Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner	2017	VLDB	5.8420545e-05
4,128	Advanced Join Strategies for Large-Scale Distributed Computation	2014	VLDB	6.4214449e-05
11,843	An Efficient MapReduce Cube Algorithm for Varied Data Distributions	2016	SIGMOD	4.1905499e-05
5,116	AdaptDB: Adaptive Partitioning for Distributed Joins	2017	VLDB	5.6805476e-05
5,309	Continuous Cloud-Scale Query Optimization and Processing	2013	VLDB	5.5714729e-05
6,671	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	4.9625353e-05
4,068	Advanced Partitioning Techniques for Massively Distributed Computation	2012	SIGMOD	6.4748133e-05