Introduction to Spark 2.0 for Database Researchers

Summary: Introduces Spark 2.0 to database researchers, focusing on Catalyst, whole-stage code generation, and hands-on use of DataFrames, SQL, streaming, and ML pipelines. Emphasizes engine internals and a practical path to hack Spark by extending the query optimizer to accelerate distributed joins. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 5270
Venue: SIGMOD
Year: 2016
Pagerank: 5.21582e-05
Overall Rank: 9,803 | 32.75%
DOI: 10.1145/2882903.2912565

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Doug Bateman (Databricks)
3. Reynold Xin (Databricks)
4. Matei Zaharia (Databricks; Massachusetts Institute of Technology)

BibTeX Citation

@inproceedings{armbrust_sigmod16,
        title = {{Introduction to Spark 2.0 for Database Researchers}},
        author = {Armbrust, Michael and Bateman, Doug and Xin, Reynold and Zaharia, Matei},
        series = {{SIGMOD} '16},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2882903.2912565},
        url = {https://dl.acm.org/doi/10.1145/2882903.2912565},
        year = {2016}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,250	Data Management in Machine Learning: Challenges, Techniques, and Systems	2017	SIGMOD	0.00011485301

Outgoing Citations (Sorted by Pagerank)

Showing 4 of 4 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
24	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00054865648
425	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00018704491
3,411	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	7.436229e-05
6,865	SparkR: Scaling R Programs with Spark	2016	SIGMOD	5.7508365e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,190	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
2	9,655	[Demo] Low-latency Spark Queries on Updatable Data	2019	SIGMOD
3	8,615	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
4	6,865	SparkR: Scaling R Programs with Spark	2016	SIGMOD
5	5,865	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB
6	6,850	Bridging the Gap Between HPC and Big Data Frameworks	2017	VLDB
7	2,594	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
8	24	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD
9	3,411	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
10	13,550	Trends and Challenges in Big Data Processing	2016	VLDB