Magnet: Push-based Shuffle Service for Large-scale Data Processing
Summary: Magnet is a push-based Spark shuffle service that merges fragmented intermediate data into large blocks and co-locates them with reducers, scaling to petabytes/day and thousands of nodes. It cuts LinkedIn production job runtimes nearly 30% while eliminating shuffle tuning. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Min Shen (LinkedIn)
- 2. Ye Zhou (LinkedIn)
- 3. Chandni Singh (LinkedIn)
BibTeX Citation
@article{shen_vldb20,
title = {{Magnet: Push-based Shuffle Service for Large-scale Data Processing}},
author = {Shen, Min and Zhou, Ye and Singh, Chandni},
journal = {PVLDB},
series = {{VLDB} '20},
volume = {13},
number = {12},
pages = {3382--3395},
doi = {10.14778/3415478.3415558},
url = {https://doi.org/10.14778/3415478.3415558},
year = {2020}
}Incoming Citations (Sorted by Pagerank)
Showing 5 of 5 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6,920 | Petabyte-Scale Row-Level Operations in Data Lakehouses | 2024 | VLDB | 5.6441822e-05 |
| 8,535 | New Query Optimization Techniques in the Spark Engine of Azure Synapse | 2022 | VLDB | 5.320973e-05 |
| 8,601 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB | 5.3051732e-05 |
| 9,943 | The Story of AWS Glue | 2023 | VLDB | 5.1043697e-05 |
| 10,309 | Intra-Query Runtime Elasticity for Cloud-Native Data Analysis | 2025 | SIGMOD | 5.0386264e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 3 of 3 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 23 | Spark SQL: Relational Data Processing in Spark | 2015 | SIGMOD | 0.00055406774 |
| 940 | Starfish: A Self-tuning System for Big Data Analytics | 2011 | CIDR | 0.00012964445 |
| 3,895 | Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE | 2019 | VLDB | 6.9393157e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 9,816 | Supporting Scalable Analytics with Latency Constraints | 2015 | VLDB |
| 2 | 3,066 | Scalable Big Graph Processing in MapReduce | 2014 | SIGMOD |
| 3 | 8,215 | Execution Primitives for Scalable Joins and Aggregations in Map Reduce | 2014 | VLDB |
| 4 | 6,996 | Bridging the Gap Between HPC and Big Data Frameworks | 2017 | VLDB |
| 5 | 2,282 | A Platform for Scalable One-Pass Analytics using MapReduce | 2011 | SIGMOD |
| 6 | 3,455 | Scaling Spark in the Real World: Performance and Usability | 2015 | VLDB |
| 7 | 9,307 | SpongeFiles: Mitigating Data Skew in MapReduce Using Distributed Memory | 2014 | SIGMOD |
| 8 | 3,895 | Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE | 2019 | VLDB |
| 9 | 4,288 | Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics | 2015 | VLDB |
| 10 | 8,601 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB |