Scaling Spark in the Real World: Performance and Usability

Summary: Real-world Spark deployment across diverse organizations via consulting and Databricks; analyzes adoption challenges and requirements. Reports usability and engine-performance improvements to Spark addressing broad real-world workloads. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: hd3583a3acad15eba
Venue: VLDB
Year: 2015
Pagerank: 7.2884813e-05
Overall Rank: 3,455 | 76.78%
DOI: 10.14778/2824032.2824058

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Tathagata Das (Databricks)
3. Aaron Davidson (Databricks)
4. Ali Ghodsi (Databricks)
5. Andrew Or (Databricks)
6. Josh Rosen (Databricks)
7. Ion Stoica (Databricks)
8. Patrick Wendell (Databricks)
9. Reynold Xin (Databricks)
10. Matei Zaharia (Databricks; Massachusetts Institute of Technology)

BibTeX Citation

@article{armbrust_vldb15,
        title = {{Scaling Spark in the Real World: Performance and Usability}},
        author = {Armbrust, Michael and Das, Tathagata and Davidson, Aaron and Ghodsi, Ali and Or, Andrew and Rosen, Josh and Stoica, Ion and Wendell, Patrick and Xin, Reynold and Zaharia, Matei},
        journal = {PVLDB},
        series = {{VLDB} '15},
        volume = {8},
        number = {12},
        pages = {1840--1851},
        doi = {10.14778/2824032.2824058},
        url = {https://doi.org/10.14778/2824032.2824058},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 9 of 9 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,895	Hyper Dimension Shuffle: Efficient Data Repartition at Petabyte Scale in SCOPE	2019	VLDB	6.9393157e-05
4,815	PolyFrame: A Retargetable Query-based Approach to Scaling Dataframes	2021	VLDB	6.3992929e-05
6,987	SparkR: Scaling R Programs with Spark	2016	SIGMOD	5.6283496e-05
7,386	Lachesis: Automatic Partitioning for UDF-Centric Analytics	2021	VLDB	5.538189e-05
9,246	DQDF: Data-Quality-Aware Dataframes	2022	VLDB	5.2056825e-05
9,321	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB	5.1943472e-05
9,969	DfAnalyzer: Runtime Dataflow Analysis of Scientific Applications using Provenance	2018	VLDB	5.1038322e-05
9,988	Introduction to Spark 2.0 for Database Researchers	2016	SIGMOD	5.0998353e-05
12,189	An Experimental Evaluation of Garbage Collectors on Big Data Applications	2019	VLDB	4.9793485e-05

Outgoing Citations (Sorted by Pagerank)

Showing 4 of 4 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
3	Pregel: A System for Large-Scale Graph Processing	2010	SIGMOD	0.0012092602
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055406774
1,868	G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data	2015	SIGMOD	9.4754064e-05
3,532	Rethinking Data-Intensive Science Using Scalable Analytics Systems	2015	SIGMOD	7.2261031e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	8,346	SparkCruise: Workload Optimization in Managed Spark Clusters at Microsoft	2021	VLDB
2	9,321	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB
3	2,635	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
4	13,623	Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries	2025	SIGMOD
5	1,123	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
6	10,927	A Decade of Apache Spark Structured Streaming: How We Evolved The Architecture To Meet Real-World Needs	2026	VLDB
7	6,987	SparkR: Scaling R Programs with Spark	2016	SIGMOD
8	9,988	Introduction to Spark 2.0 for Database Researchers	2016	SIGMOD
9	6,996	Bridging the Gap Between HPC and Big Data Frameworks	2017	VLDB
10	13,863	Trends and Challenges in Big Data Processing	2016	VLDB