Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics
Summary: Profiling-driven, architecture-level comparison of MapReduce and Spark across shuffle, execution model, and caching, with tools linking task plans to resource use. Spark is ~2.5x–5x faster on WordCount, k-means, and PageRank; gains from hash-based aggregation and RDD caching, while MapReduce wins Sort via shuffle efficiency. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Juwei Shi (IBM Research - China)
- 2. Yunjie Qiu (Renmin University of China)
- 3. Umar Farooq Minhas (IBM)
- 4. Limei Jiao (Renmin University of China)
- 5. Chen Wang (Tsinghua University)
- 6. Berthold Reinwald (IBM)
- 7. Fatma Özcan (IBM)
BibTeX Citation
@article{shi_vldb15,
title = {{Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics}},
author = {Shi, Juwei and Qiu, Yunjie and Minhas, Umar Farooq and Jiao, Limei and Wang, Chen and Reinwald, Berthold and Özcan, Fatma},
journal = {PVLDB},
series = {{VLDB} '15},
volume = {8},
number = {13},
doi = {10.14778/2831360.2831365},
url = {https://doi.org/10.14778/2831360.2831365},
year = {2015}
}Incoming Citations (Sorted by Pagerank)
Showing 7 of 7 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 4,308 | Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads | 2017 | VLDB | 6.6735191e-05 |
| 5,408 | Lifetime-Based Memory Management for Distributed Data Processing Systems | 2016 | VLDB | 6.1415736e-05 |
| 7,386 | Lachesis: Automatic Partitioning for UDF-Centric Analytics | 2021 | VLDB | 5.538189e-05 |
| 9,748 | PlinyCompute: A Platform for High-Performance, Distributed, Data-Intensive Tool Development | 2018 | SIGMOD | 5.1349531e-05 |
| 11,932 | Accelerating Complex Analytics using Speculation | 2021 | CIDR | 4.9793485e-05 |
| 12,189 | An Experimental Evaluation of Garbage Collectors on Big Data Applications | 2019 | VLDB | 4.9793485e-05 |
| 12,302 | Query-able Kafka: An agile data analytics pipeline for mobile wireless networks | 2017 | VLDB | 4.9793485e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 3 of 3 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 3 | Pregel: A System for Large-Scale Graph Processing | 2010 | SIGMOD | 0.0012092602 |
| 753 | Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs | 2011 | VLDB | 0.00014237583 |
| 7,632 | MRTuner: A Toolkit to Enable Holistic Optimization for MapReduce Jobs | 2014 | VLDB | 5.4793438e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 7,051 | Efficient In-memory Data Management: An Analysis | 2014 | VLDB |
| 2 | 8,704 | sPCA: Scalable Principal Component Analysis for Big Data on Distributed Platforms | 2015 | SIGMOD |
| 3 | 753 | Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs | 2011 | VLDB |
| 4 | 5,504 | Magnet: Push-based Shuffle Service for Large-scale Data Processing | 2020 | VLDB |
| 5 | 3,455 | Scaling Spark in the Real World: Performance and Usability | 2015 | VLDB |
| 6 | 3,066 | Scalable Big Graph Processing in MapReduce | 2014 | SIGMOD |
| 7 | 2,282 | A Platform for Scalable One-Pass Analytics using MapReduce | 2011 | SIGMOD |
| 8 | 6,996 | Bridging the Gap Between HPC and Big Data Frameworks | 2017 | VLDB |
| 9 | 43 | A Comparison of Approaches to Large-Scale Data Analysis | 2009 | SIGMOD |
| 10 | 1,466 | The Performance of MapReduce: An In-depth Study | 2010 | VLDB |