Scaling Spark in the Real World: Performance and Usability
Summary: Real-world Spark deployment across diverse organizations via consulting and Databricks; analyzes adoption challenges and requirements. Reports usability and engine-performance improvements to Spark addressing broad real-world workloads. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Michael Armbrust (Databricks)
- 2. Tathagata Das (Databricks)
- 3. Aaron Davidson (Databricks)
- 4. Ali Ghodsi (Databricks)
- 5. Andrew Or (Databricks)
- 6. Josh Rosen (Databricks)
- 7. Ion Stoica (Databricks)
- 8. Patrick Wendell (Databricks)
- 9. Reynold Xin (Databricks)
- 10. Matei Zaharia (Databricks; Massachusetts Institute of Technology)
BibTeX Citation
@article{armbrust_vldb15,
title = {{Scaling Spark in the Real World: Performance and Usability}},
author = {Armbrust, Michael and Das, Tathagata and Davidson, Aaron and Ghodsi, Ali and Or, Andrew and Rosen, Josh and Stoica, Ion and Wendell, Patrick and Xin, Reynold and Zaharia, Matei},
journal = {PVLDB},
series = {{VLDB} '15},
volume = {8},
number = {12},
pages = {1840--1851},
doi = {10.14778/2824032.2824058},
url = {https://doi.org/10.14778/2824032.2824058},
year = {2015}
}Incoming Citations (Sorted by Pagerank)
Showing 9 of 9 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 3,964 | Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE | 2019 | VLDB | 6.9855158e-05 |
| 4,728 | PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes | 2021 | VLDB | 6.5360246e-05 |
| 6,865 | SparkR: Scaling R Programs with Spark | 2016 | SIGMOD | 5.7508365e-05 |
| 7,311 | Lachesis: Automatic Partitioning for UDF-Centric Analytics | 2021 | VLDB | 5.6491618e-05 |
| 9,069 | DQDF: Data-Quality-Aware Dataframes | 2022 | VLDB | 5.3251649e-05 |
| 9,167 | Dynamic Speculative Optimizations for SQL Compilation in Apache Spark | 2020 | VLDB | 5.3099969e-05 |
| 9,803 | Introduction to Spark 2.0 for Database Researchers | 2016 | SIGMOD | 5.21582e-05 |
| 11,889 | An Experimental Evaluation of Garbage Collectors on Big Data Applications | 2019 | VLDB | 5.093636e-05 |
| 11,949 | DfAnalyzer: Runtime Dataflow Analysis of Scientific Applications using Provenance | 2018 | VLDB | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 4 of 4 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 3 | Pregel: A System for Large-Scale Graph Processing | 2010 | SIGMOD | 0.0012250108 |
| 24 | Spark SQL: Relational Data Processing in Spark | 2015 | SIGMOD | 0.00054865648 |
| 1,827 | G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data | 2015 | SIGMOD | 9.6690206e-05 |
| 3,552 | Rethinking Data-Intensive Science Using Scalable Analytics Systems | 2015 | SIGMOD | 7.3188008e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 8,615 | A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning | 2024 | VLDB |
| 2 | 9,167 | Dynamic Speculative Optimizations for SQL Compilation in Apache Spark | 2020 | VLDB |
| 3 | 8,175 | SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft | 2021 | VLDB |
| 4 | 2,594 | Big Data Analytics with Datalog Queries on Spark | 2016 | SIGMOD |
| 5 | 13,301 | Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries | 2025 | SIGMOD |
| 6 | 1,190 | Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark | 2018 | SIGMOD |
| 7 | 6,865 | SparkR: Scaling R Programs with Spark | 2016 | SIGMOD |
| 8 | 9,803 | Introduction to Spark 2.0 for Database Researchers | 2016 | SIGMOD |
| 9 | 6,850 | Bridging the Gap Between HPC and Big Data Frameworks | 2017 | VLDB |
| 10 | 13,550 | Trends and Challenges in Big Data Processing | 2016 | VLDB |