DBScholar

Back to papers

Spark SQL: Relational Data Processing in Spark

Summary: Relational processing integrated into Spark via DataFrame API, unifying SQL queries with Spark's functional workflow. Catalyst, a Scala-based extensible optimizer, enables composable rules, code generation, JSON schema inference, and federation to databases. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h2541b5993b725084
Venue
SIGMOD
Year
2015
Pagerank
0.00055406774
Overall Rank
23 | 99.85%
DOI
10.1145/2723372.2742797

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{armbrust_sigmod15,
        title = {{Spark SQL: Relational Data Processing in Spark}},
        author = {Armbrust, Michael and Xin, Reynold S. and Lian, Cheng and Huai, Yin and Liu, Davies and Bradley, Joseph K. and Meng, Xiangrui and Kaftan, Tomer and Franklin, Michael J. and Ghodsi, Ali and Zaharia, Matei},
        series = {{SIGMOD} '15},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/2723372.2742797},
        url = {https://dl.acm.org/doi/10.1145/2723372.2742797},
        year = {2015}
}

Incoming Citations (Sorted by Pagerank)

Showing 50 of 219 citing papers.

Rank Citing Paper Year Venue Pagerank
281 Accelerating Machine Learning Inference with Probabilistic Predicates 2018 SIGMOD 0.00022295232
341 AnalyticDB-V: A Hybrid Analytical Engine Towards Query Fusion for Structured and Unstructured Data 2020 VLDB 0.00020539791
379 Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources 2018 SIGMOD 0.00019514689
459 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017856221
512 NeuroCard: One Cardinality Estimator for All Tables 2021 VLDB 0.00017050173
596 Wander Join: Online Aggregation via Random Walks 2016 SIGMOD 0.00015785583
795 Random Sampling over Joins Revisited 2018 SIGMOD 0.00013938779
840 Quickr: Lazily Approximating Complex AdHoc Queries in BigData Clusters 2016 SIGMOD 0.0001354605
950 Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics 2021 CIDR 0.00012895553
1,082 Approximate Query Processing: No Silver Bullet 2017 SIGMOD 0.00012122749
1,123 Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark 2018 SIGMOD 0.0001193233
1,178 Simba: Efficient In-Memory Spatial Analytics 2016 SIGMOD 0.00011632691
1,193 Weld: A Common Runtime for High Performance Data Analytics 2017 CIDR 0.0001158809
1,308 Automating Large-Scale Data Quality Verification 2018 VLDB 0.0001107886
1,321 Hybrid Transactional/Analytical Processing: A Survey 2017 SIGMOD 0.00011040617
1,369 Towards Scalable Dataframe Systems 2020 VLDB 0.00010899832
1,423 Procella: Unifying serving and analytical data at YouTube 2019 VLDB 0.00010719956
1,487 Photon: A Fast Query Engine for Lakehouse Systems 2022 SIGMOD 0.00010521722
1,564 Titian: Data Provenance Support in Spark 2016 VLDB 0.00010222394
1,568 HELIX: Holistic Optimization for Accelerating Iterative Machine Learning 2019 VLDB 0.0001021302
1,680 Axiomatic Foundations and Algorithms for Deciding Semantic Equivalences of SQL Queries 2018 VLDB 9.9031411e-05
1,742 How to Architect a Query Compiler 2016 SIGMOD 9.7378418e-05
1,790 POLARIS: The Distributed SQL Engine in Azure Synapse 2020 VLDB 9.6272006e-05
1,803 Tuplex: Data Science in Python at Native Code Speed 2021 SIGMOD 9.6068397e-05
1,829 DBEst: Revisiting Approximate Query Processing Engines with Machine Learning Models 2019 SIGMOD 9.5510333e-05
1,911 ModelarDB: Modular Model-Based Time Series Management with Spark and Cassandra 2018 VLDB 9.3912399e-05
2,004 FLAT: Fast, Lightweight and Accurate Method for Cardinality Estimation 2021 VLDB 9.2065719e-05
2,027 Database Learning: Toward a Database that Becomes Smarter Every Time 2017 SIGMOD 9.1618139e-05
2,101 DUALSIM: Parallel Subgraph Enumeration in a Massive Graph on a Single Machine 2016 SIGMOD 9.0476814e-05
2,137 Quickstep: A Data Platform Based on the Scaling-Up Approach 2018 VLDB 8.9777553e-05
2,160 DIFF: A Relational Interface for Large-Scale Data Explanation 2019 VLDB 8.9364035e-05
2,196 DITA: Distributed In-Memory Trajectory Analytics 2018 SIGMOD 8.8780862e-05
2,249 Evaluating End-to-End Optimization for Data Analytics Applications in Weld 2018 VLDB 8.7549752e-05
2,347 Filter Before You Parse: Faster Analytics on Raw Data with Sparser 2018 VLDB 8.6012176e-05
2,400 Declarative Recursive Computation on an RDBMS or, Why You Should Use a Database For Distributed Machine Learning 2019 VLDB 8.5213814e-05
2,419 How to Architect a Query Compiler, Revisited 2018 SIGMOD 8.4923505e-05
2,436 Mison: A Fast JSON Parser for Data Analytics 2017 VLDB 8.4708399e-05
2,484 AnalyticDB: Real-time OLAP Database System at Alibaba Cloud 2019 VLDB 8.3993882e-05
2,550 SQLShare: Results from a Multi-Year SQL-as-a-Service Experiment 2016 SIGMOD 8.3121033e-05
2,635 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD 8.1965216e-05
2,662 End-to-end Optimization of Machine Learning Prediction Queries 2022 SIGMOD 8.1596229e-05
2,735 Speculative Distributed CSV Data Parsing for Big Data Analytics 2019 SIGMOD 8.0761736e-05
2,778 AIDA - Abstraction for Advanced In-Database Analytics 2018 VLDB 8.0299506e-05
2,812 OceanBase: A 707 Million tpmC Distributed Relational Database System 2022 VLDB 7.9811649e-05
2,915 F1 Query: Declarative Querying at Scale 2018 VLDB 7.8616593e-05
2,938 RaSQL: Greater Power and Performance for Big Data Analytics with Recursive-aggregate-SQL on Spark 2019 SIGMOD 7.8355984e-05
2,997 F1 Lightning: HTAP as a Service 2020 VLDB 7.7689996e-05
3,033 Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems 2018 VLDB 7.7337998e-05
3,073 Pushing Data-Induced Predicates Through Joins in Big-Data Clusters 2020 VLDB 7.6777283e-05
3,081 Skipping-oriented Partitioning for Columnar Layouts 2017 VLDB 7.6653727e-05
Previous Page 1 / 5 Next

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
6 Pig Latin: A Not-So-Foreign Language for Data Processing 2008 SIGMOD 0.001052036
22 Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud 2012 VLDB 0.00055962491
43 A Comparison of Approaches to Large-Scale Data Analysis 2009 SIGMOD 0.00045546775
49 Dremel: Interactive Analysis of Web-Scale Datasets 2010 VLDB 0.00043160717
118 The EXODUS Optimizer Generator 1987 SIGMOD 0.00031392616
154 MAD Skills: New Analysis Practices for Big Data 2009 VLDB 0.00028579704
327 Impala: A Modern, Open-Source SQL Engine for Hadoop 2015 CIDR 0.0002095191
432 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018339357
495 Building Efficient Query Engines in a High-Level Language 2014 VLDB 0.00017370758
1,258 Extracting Schema from Semistructured Data 1998 SIGMOD 0.00011306716
1,743 Sinew: A SQL System for Multi-Structured Data 2014 SIGMOD 9.7368807e-05
1,810 Distributed Data-Parallel Computing Using a High-Level Programming Language 2009 SIGMOD 9.5892079e-05
1,868 G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data 2015 SIGMOD 9.4754064e-05
2,797 Inferring XML Schema Definitions from XML Data 2007 VLDB 7.9969811e-05
3,532 Rethinking Data-Intensive Science Using Scalable Analytics Systems 2015 SIGMOD 7.2261031e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 3,455 Scaling Spark in the Real World: Performance and Usability 2015 VLDB
2 432 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD
3 415 SystemML: Declarative Machine Learning on Spark 2016 VLDB
4 2,635 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD
5 12,074 RASQL: A Powerful Language and its System for Big Data Applications 2020 SIGMOD
6 1,123 Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark 2018 SIGMOD
7 6,987 SparkR: Scaling R Programs with Spark 2016 SIGMOD
8 9,988 Introduction to Spark 2.0 for Database Researchers 2016 SIGMOD
9 9,321 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB
10 2,938 RaSQL: Greater Power and Performance for Big Data Analytics with Recursive-aggregate-SQL on Spark 2019 SIGMOD