Spark SQL: Relational Data Processing in Spark

Summary: Relational processing integrated into Spark via DataFrame API, unifying SQL queries with Spark's functional workflow. Catalyst, a Scala-based extensible optimizer, enables composable rules, code generation, JSON schema inference, and federation to databases. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h2541b5993b725084
Venue: SIGMOD
Year: 2015
Pagerank: 0.00055406774
Overall Rank: 23 | 99.85%
DOI: 10.1145/2723372.2742797

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Reynold S. Xin (Databricks)
3. Cheng Lian (Databricks)
4. Yin Huai (Databricks)
5. Davies Liu (Databricks)
6. Joseph K. Bradley (Databricks)
7. Xiangrui Meng (Databricks)
8. Tomer Kaftan (University of California Berkeley)
9. Michael J. Franklin (Databricks; University of California Berkeley)
10. Ali Ghodsi (Databricks)
11. Matei Zaharia (Databricks; Massachusetts Institute of Technology)

BibTeX Citation

@inproceedings{armbrust_sigmod15,
        title = {{Spark SQL: Relational Data Processing in Spark}},
        author = {Armbrust, Michael and Xin, Reynold S. and Lian, Cheng and Huai, Yin and Liu, Davies and Bradley, Joseph K. and Meng, Xiangrui and Kaftan, Tomer and Franklin, Michael J. and Ghodsi, Ali and Zaharia, Matei},
        series = {{SIGMOD} '15},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2723372.2742797},
        url = {https://dl.acm.org/doi/10.1145/2723372.2742797},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 19 of 219 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
11,676	Templating Shuffles	2023	CIDR	4.9793485e-05
11,708	dsJSON: A Distributed SQL JSON Processor	2023	SIGMOD	4.9793485e-05
11,736	TEE-based General-purpose Computational Backend for Secure Delegated Data Processing	2023	SIGMOD	4.9793485e-05
11,777	Anser: Adaptive Information Sharing Framework of AnalyticDB	2023	VLDB	4.9793485e-05
11,803	XDB in Action: Decentralized Cross-Database Query Processing for Black-Box DBMSes	2023	VLDB	4.9793485e-05
11,848	Juggler: Autonomous Cost Optimization and Performance Prediction of Big Data Applications	2022	SIGMOD	4.9793485e-05
11,984	Vertex-centric Parallel Computation of SQL Queries	2021	SIGMOD	4.9793485e-05
11,990	Real-time Data Infrastructure at Uber	2021	SIGMOD	4.9793485e-05
12,023	AnyOLAP: Analytical Processing of Arbitrary Data-Intensive Applications without ETL	2021	VLDB	4.9793485e-05
12,031	Fangorn: Adaptive Execution Framework for Heterogeneous Workloads on Shared Clusters	2021	VLDB	4.9793485e-05
12,074	RASQL: A Powerful Language and its System for Big Data Applications	2020	SIGMOD	4.9793485e-05
12,167	Block as a Value for SQL over NoSQL	2019	VLDB	4.9793485e-05
12,185	Integration of Large-Scale Data Processing Systems and Traditional Parallel Database Technology	2019	VLDB	4.9793485e-05
12,189	An Experimental Evaluation of Garbage Collectors on Big Data Applications	2019	VLDB	4.9793485e-05
12,253	An Authorization Model for Multi Provider Queries	2018	VLDB	4.9793485e-05
12,257	Effective Temporal Dependence Discovery in Time Series Data	2018	VLDB	4.9793485e-05
12,277	Query Processing Techniques for Big Spatial-Keyword Data	2017	SIGMOD	4.9793485e-05
12,438	Tutorial: SQL-on-Hadoop Systems	2015	VLDB	4.9793485e-05
13,623	Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries	2025	SIGMOD	-

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.001052036
22	Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud	2012	VLDB	0.00055962491
43	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD	0.00045546775
49	Dremel: Interactive Analysis of Web-Scale Datasets	2010	VLDB	0.00043160717
118	The EXODUS Optimizer Generator	1987	SIGMOD	0.00031392616
154	MAD Skills: New Analysis Practices for Big Data	2009	VLDB	0.00028579704
327	Impala: A Modern, Open-Source SQL Engine for Hadoop	2015	CIDR	0.0002095191
432	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00018339357
495	Building Efficient Query Engines in a High-Level Language	2014	VLDB	0.00017370758
1,258	Extracting Schema from Semistructured Data	1998	SIGMOD	0.00011306716
1,743	Sinew: A SQL System for Multi-Structured Data	2014	SIGMOD	9.7368807e-05
1,810	Distributed Data-Parallel Computing Using a High-Level Programming Language	2009	SIGMOD	9.5892079e-05
1,868	G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data	2015	SIGMOD	9.4754064e-05
2,797	Inferring XML Schema Definitions from XML Data	2007	VLDB	7.9969811e-05
3,532	Rethinking Data-Intensive Science Using Scalable Analytics Systems	2015	SIGMOD	7.2261031e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,455	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
2	432	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD
3	415	SystemML: Declarative Machine Learning on Spark	2016	VLDB
4	2,635	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
5	12,074	RASQL: A Powerful Language and its System for Big Data Applications	2020	SIGMOD
6	1,123	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
7	6,987	SparkR: Scaling R Programs with Spark	2016	SIGMOD
8	9,988	Introduction to Spark 2.0 for Database Researchers	2016	SIGMOD
9	9,321	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB
10	2,938	RaSQL: Greater Power and Performance for Big Data Analytics with Recursive-aggregate-SQL on Spark	2019	SIGMOD