SpongeFiles: Mitigating Data Skew in MapReduce Using Distributed Memory
Summary: Introduces SpongeFiles, a distributed-memory abstraction for MapReduce that stores spilled data as a logical byte array across memory and disk, reducing data skew. Nearest-capacity routing lets idle nodes tap neighbor memory/disk, yielding up to 55% overall speedup (85% under contention) on Hadoop/Pig. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Khaled Elmeleegy (Turn Inc.; Yahoo)
- 2. Christopher Olston (Google; Yahoo)
- 3. Benjamin Reed (Meta; Yahoo)
BibTeX Citation
@inproceedings{elmeleegy_sigmod14,
title = {{SpongeFiles: Mitigating Data Skew in MapReduce Using Distributed Memory}},
author = {Elmeleegy, Khaled and Olston, Christopher and Reed, Benjamin},
series = {{SIGMOD} '14},
booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
publisher = {Association for Computing Machinery},
doi = {10.1145/2588555.2595634},
url = {https://dl.acm.org/doi/10.1145/2588555.2595634},
year = {2014}
}Incoming Citations (Sorted by Pagerank)
Showing 1 of 1 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6,734 | Liquid: Unifying Nearline and Offline Big Data Integration | 2015 | CIDR | 5.7879923e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 9 of 9 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6 | Pig Latin: A Not-So-Foreign Language for Data Processing | 2008 | SIGMOD | 0.0010686205 |
| 47 | PNUTS: Yahoo!'s Hosted Data Serving Platform | 2008 | VLDB | 0.00044503718 |
| 471 | Skew-Aware Automatic Database Partitioning in Shared-Nothing, Parallel OLTP Systems | 2012 | SIGMOD | 0.0001793564 |
| 481 | Practical Skew Handling in Parallel Joins | 1992 | VLDB | 0.00017780716 |
| 642 | Building a High-Level Dataflow System on top of Map-Reduce: The Pig Experience | 2009 | VLDB | 0.00015395331 |
| 811 | A Taxonomy and Performance Model of Data Skew Effects in Parallel Joins | 1991 | VLDB | 0.00013859761 |
| 1,236 | Handling Data Skew in Multiprocessor Database Computers Using Partition Tuning | 1991 | VLDB | 0.00011548179 |
| 1,319 | SkewTune: Mitigating Skew in MapReduce Applications | 2012 | SIGMOD | 0.00011175005 |
| 8,455 | Piranha: Optimizing Short Jobs in Hadoop | 2013 | VLDB | 5.4226e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 5,390 | Magnet: Push-based Shuffle Service for Large-scale Data Processing | 2020 | VLDB |
| 2 | 3,466 | Advanced Partitioning Techniques for Massively Distributed Computation | 2012 | SIGMOD |
| 3 | 2,849 | Column-Oriented Storage Techniques for MapReduce | 2011 | VLDB |
| 4 | 7,932 | Hone: “Scaling Down” Hadoop on Shared-Memory Systems | 2013 | VLDB |
| 5 | 1,883 | ReStore: Reusing Results of MapReduce Jobs | 2012 | VLDB |
| 6 | 12,336 | SkewTune in Action: Mitigating Skew in MapReduce Applications | 2012 | VLDB |
| 7 | 2,265 | A Platform for Scalable One-Pass Analytics using MapReduce | 2011 | SIGMOD |
| 8 | 12,131 | FP-Hadoop: Efficient Execution of Parallel Jobs Over Skewed Data | 2015 | VLDB |
| 9 | 1,319 | SkewTune: Mitigating Skew in MapReduce Applications | 2012 | SIGMOD |
| 10 | 12,036 | An Efficient MapReduce Cube Algorithm for Varied Data Distributions | 2016 | SIGMOD |