Introduction to Spark 2.0 for Database Researchers

Summary: Introduces Spark 2.0 to database researchers, focusing on Catalyst, whole-stage code generation, and hands-on use of DataFrames, SQL, streaming, and ML pipelines. Emphasizes engine internals and a practical path to hack Spark by extending the query optimizer to accelerate distributed joins. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h1fc19baaecf787c1
Venue: SIGMOD
Year: 2016
Pagerank: 5.082155e-05
Overall Rank: 10,020 | 32.84%
DOI: 10.1145/2882903.2912565

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Doug Bateman (Databricks)
3. Reynold Xin (Databricks)
4. Matei Zaharia (Databricks; Massachusetts Institute of Technology)

BibTeX Citation

@inproceedings{armbrust_sigmod16,
        title = {{Introduction to Spark 2.0 for Database Researchers}},
        author = {Armbrust, Michael and Bateman, Doug and Xin, Reynold and Zaharia, Matei},
        series = {{SIGMOD} '16},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2882903.2912565},
        url = {https://dl.acm.org/doi/10.1145/2882903.2912565},
        year = {2016}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,226	Data Management in Machine Learning: Challenges, Techniques, and Systems	2017	SIGMOD	0.00011432354

Outgoing Citations (Sorted by Pagerank)

Showing 4 of 4 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055523999
436	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00018274105
3,466	Scaling Spark in the Real World: Performance and Usability	2015	VLDB	7.2609893e-05
6,996	SparkR: Scaling R Programs with Spark	2016	SIGMOD	5.6092026e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,128	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
2	9,873	[Demo] Low-latency Spark Queries on Updatable Data	2019	SIGMOD
3	8,026	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
4	6,996	SparkR: Scaling R Programs with Spark	2016	SIGMOD
5	5,979	Speedup Your Analytics: Automatic Parameter Tuning for Databases and Big Data Systems	2019	VLDB
6	7,016	Bridging the Gap Between HPC and Big Data Frameworks	2017	VLDB
7	2,644	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
8	23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD
9	3,466	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
10	13,908	Trends and Challenges in Big Data Processing	2016	VLDB