Spark SQL: Relational Data Processing in Spark

Summary: Relational processing integrated into Spark via DataFrame API, unifying SQL queries with Spark's functional workflow. Catalyst, a Scala-based extensible optimizer, enables composable rules, code generation, JSON schema inference, and federation to databases. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h2541b5993b725084
Venue: SIGMOD
Year: 2015
Pagerank: 0.00055523999
Overall Rank: 23 | 99.85%
DOI: 10.1145/2723372.2742797

Incoming Non-self Citations Over Time

Authors

1. Michael Armbrust (Databricks)
2. Reynold S. Xin (Databricks)
3. Cheng Lian (Databricks)
4. Yin Huai (Databricks)
5. Davies Liu (Databricks)
6. Joseph K. Bradley (Databricks)
7. Xiangrui Meng (Databricks)
8. Tomer Kaftan (University of California Berkeley)
9. Michael J. Franklin (Databricks; University of California Berkeley)
10. Ali Ghodsi (Databricks)
11. Matei Zaharia (Databricks; Massachusetts Institute of Technology)

BibTeX Citation

@inproceedings{armbrust_sigmod15,
        title = {{Spark SQL: Relational Data Processing in Spark}},
        author = {Armbrust, Michael and Xin, Reynold S. and Lian, Cheng and Huai, Yin and Liu, Davies and Bradley, Joseph K. and Meng, Xiangrui and Kaftan, Tomer and Franklin, Michael J. and Ghodsi, Ali and Zaharia, Matei},
        series = {{SIGMOD} '15},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2723372.2742797},
        url = {https://dl.acm.org/doi/10.1145/2723372.2742797},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 21 of 221 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
11,664	Large-Scale Metric Computation in Online Controlled Experiment Platform	2024	VLDB	4.9605343e-05
11,722	Raising the Level of Abstraction for Time-State Analytics With the Timeline Framework	2023	CIDR	4.9605343e-05
11,728	Templating Shuffles	2023	CIDR	4.9605343e-05
11,760	dsJSON: A Distributed SQL JSON Processor	2023	SIGMOD	4.9605343e-05
11,786	TEE-based General-purpose Computational Backend for Secure Delegated Data Processing	2023	SIGMOD	4.9605343e-05
11,827	Anser: Adaptive Information Sharing Framework of AnalyticDB	2023	VLDB	4.9605343e-05
11,853	XDB in Action: Decentralized Cross-Database Query Processing for Black-Box DBMSes	2023	VLDB	4.9605343e-05
11,898	Juggler: Autonomous Cost Optimization and Performance Prediction of Big Data Applications	2022	SIGMOD	4.9605343e-05
12,034	Vertex-centric Parallel Computation of SQL Queries	2021	SIGMOD	4.9605343e-05
12,040	Real-time Data Infrastructure at Uber	2021	SIGMOD	4.9605343e-05
12,073	AnyOLAP: Analytical Processing of Arbitrary Data-Intensive Applications without ETL	2021	VLDB	4.9605343e-05
12,081	Fangorn: Adaptive Execution Framework for Heterogeneous Workloads on Shared Clusters	2021	VLDB	4.9605343e-05
12,124	RASQL: A Powerful Language and its System for Big Data Applications	2020	SIGMOD	4.9605343e-05
12,216	Block as a Value for SQL over NoSQL	2019	VLDB	4.9605343e-05
12,234	Integration of Large-Scale Data Processing Systems and Traditional Parallel Database Technology	2019	VLDB	4.9605343e-05
12,238	An Experimental Evaluation of Garbage Collectors on Big Data Applications	2019	VLDB	4.9605343e-05
12,302	An Authorization Model for Multi Provider Queries	2018	VLDB	4.9605343e-05
12,306	Effective Temporal Dependence Discovery in Time Series Data	2018	VLDB	4.9605343e-05
12,326	Query Processing Techniques for Big Spatial-Keyword Data	2017	SIGMOD	4.9605343e-05
12,487	Tutorial: SQL-on-Hadoop Systems	2015	VLDB	4.9605343e-05
13,670	Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries	2025	SIGMOD	-

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
6	Pig Latin: A Not-So-Foreign Language for Data Processing	2008	SIGMOD	0.0010494385
22	Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud	2012	VLDB	0.00056028465
43	A Comparison of Approaches to Large-Scale Data Analysis	2009	SIGMOD	0.00045453255
49	Dremel: Interactive Analysis of Web-Scale Datasets	2010	VLDB	0.00043085722
118	The EXODUS Optimizer Generator	1987	SIGMOD	0.00031320165
154	MAD Skills: New Analysis Practices for Big Data	2009	VLDB	0.00028524078
327	Impala: A Modern, Open-Source SQL Engine for Hadoop	2015	CIDR	0.00020904249
436	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD	0.00018274105
498	Building Efficient Query Engines in a High-Level Language	2014	VLDB	0.00017323427
1,262	Extracting Schema from Semistructured Data	1998	SIGMOD	0.00011277501
1,753	Sinew: A SQL System for Multi-Structured Data	2014	SIGMOD	9.7119095e-05
1,814	Distributed Data-Parallel Computing Using a High-Level Programming Language	2009	SIGMOD	9.5659163e-05
1,874	G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data	2015	SIGMOD	9.4470874e-05
2,800	Inferring XML Schema Definitions from XML Data	2007	VLDB	7.9784397e-05
3,537	Rethinking Data-Intensive Science Using Scalable Analytics Systems	2015	SIGMOD	7.2010811e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	3,466	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
2	436	Shark: SQL and Rich Analytics at Scale	2013	SIGMOD
3	415	SystemML: Declarative Machine Learning on Spark	2016	VLDB
4	2,644	Big Data Analytics with Datalog Queries on Spark	2016	SIGMOD
5	12,124	RASQL: A Powerful Language and its System for Big Data Applications	2020	SIGMOD
6	1,128	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
7	6,996	SparkR: Scaling R Programs with Spark	2016	SIGMOD
8	10,020	Introduction to Spark 2.0 for Database Researchers	2016	SIGMOD
9	9,356	Dynamic Speculative Optimizations for SQL Compilation in Apache Spark	2020	VLDB
10	2,953	RaSQL: Greater Power and Performance for Big Data Analytics with Recursive-aggregate-SQL on Spark	2019	SIGMOD