Magnet: Push-based Shuffle Service for Large-scale Data Processing
Summary: Magnet is a push-based Spark shuffle service that merges fragmented intermediate data into large blocks and co-locates them with reducers, scaling to petabytes/day and thousands of nodes. It cuts LinkedIn production job runtimes nearly 30% while eliminating shuffle tuning. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Min Shen (LinkedIn)
- 2. Ye Zhou (LinkedIn)
- 3. Chandni Singh (LinkedIn)
BibTeX Citation
@article{shen_vldb20,
title = {{Magnet: Push-based Shuffle Service for Large-scale Data Processing}},
author = {Shen, Min and Zhou, Ye and Singh, Chandni},
journal = {PVLDB},
series = {{VLDB} '20},
volume = {13},
number = {12},
pages = {3382--3395},
doi = {10.14778/3415478.3415558},
url = {https://doi.org/10.14778/3415478.3415558},
year = {2020}
}Incoming Citations (Sorted by Pagerank)
Showing 5 of 5 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 7,780 | Petabyte-Scale Row-Level Operations in Data Lakehouses | 2024 | VLDB | 5.5457298e-05 |
| 8,439 | New Query Optimization Techniques in the Spark Engine of Azure Synapse | 2022 | VLDB | 5.4248071e-05 |
| 8,457 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB | 5.4217837e-05 |
| 9,797 | The Story of AWS Glue | 2023 | VLDB | 5.21848e-05 |
| 10,768 | Intra-Query Runtime Elasticity for Cloud-Native Data Analysis | 2025 | SIGMOD | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 3 of 3 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 24 | Spark SQL: Relational Data Processing in Spark | 2015 | SIGMOD | 0.00054865648 |
| 923 | Starfish: A Self-tuning System for Big Data Analytics | 2011 | CIDR | 0.00013189886 |
| 3,964 | Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE | 2019 | VLDB | 6.9855158e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 4,480 | LocationSpark: A Distributed In-Memory Data Management System for Big Spatial Data | 2016 | VLDB |
| 2 | 8,069 | Execution Primitives for Scalable Joins and Aggregations in Map Reduce | 2014 | VLDB |
| 3 | 3,008 | Scalable Big Graph Processing in MapReduce | 2014 | SIGMOD |
| 4 | 6,850 | Bridging the Gap Between HPC and Big Data Frameworks | 2017 | VLDB |
| 5 | 3,411 | Scaling Spark in the Real World: Performance and Usability | 2015 | VLDB |
| 6 | 2,265 | A Platform for Scalable One-Pass Analytics using MapReduce | 2011 | SIGMOD |
| 7 | 9,137 | SpongeFiles: Mitigating Data Skew in MapReduce Using Distributed Memory | 2014 | SIGMOD |
| 8 | 3,964 | Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE | 2019 | VLDB |
| 9 | 4,208 | Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics | 2015 | VLDB |
| 10 | 8,457 | Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance | 2024 | VLDB |