Database Paper Browser

Back to papers

Spark SQL: Relational Data Processing in Spark

Summary: Relational processing integrated into Spark via DataFrame API, unifying SQL queries with Spark's functional workflow. Catalyst, a Scala-based extensible optimizer, enables composable rules, code generation, JSON schema inference, and federation to databases. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5023
Venue
SIGMOD
Year
2015
Pagerank
0.00055280049
Overall Rank
25 | 99.83%
DOI
10.1145/2723372.2742797

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 50 of 206 citing papers.

Rank Citing Paper Year Venue Pagerank
9,283 Language-Agnostic Integrated Queries in a Managed Polyglot Runtime 2021 VLDB 5.3548287e-05
9,298 In-Browser Interactive SQL Analytics with Afterburner 2017 SIGMOD 5.351019e-05
9,337 SODA: A Set of Fast Oblivious Algorithms in Distributed Secure Data Analytics 2023 VLDB 5.3449422e-05
9,340 Parallel Query Processing: To Separate Communication from Computation 2022 SIGMOD 5.3449422e-05
9,342 PlinyCompute: A Platform for High-Performance, Distributed, Data-Intensive Tool Development 2018 SIGMOD 5.3449422e-05
9,361 GIO: Generating Efficient Matrix and Frame Readers for Custom Data Formats by Example 2023 SIGMOD 5.3422031e-05
9,415 TreeToaster: Towards an IVM-Optimized Compiler 2021 SIGMOD 5.3341661e-05
9,437 Rock: Cleaning Data by Embedding ML in Logic Rules 2024 SIGMOD 5.3331413e-05
9,510 [Demo] Low-latency Spark Queries on Updatable Data 2019 SIGMOD 5.32339e-05
9,595 Polyglot Data Management: State of the Art & Open Challenges 2022 VLDB 5.3086438e-05
9,678 Introduction to Spark 2.0 for Database Researchers 2016 SIGMOD 5.293328e-05
9,689 Evaluating Query Languages and Systems for High-Energy Physics Data 2022 VLDB 5.2917615e-05
9,706 Tuplex: Robust, Efficient Analytics When Python Rules 2019 VLDB 5.2889576e-05
9,846 Saving Money for Analytical Workloads in the Cloud 2024 VLDB 5.2617062e-05
9,923 Chukonu: A Fully-Featured High-Performance Big Data Framework that Integrates a Native Compute Engine into Spark 2022 VLDB 5.2412669e-05
10,144 Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization 2026 SIGMOD 5.1725247e-05
10,257 SIDLE: Tree-structure Aware Indexes for CXL-based Heterogeneous Memory 2026 VLDB 5.1725247e-05
10,396 Optimizing Block Skipping for High-Dimensional Data with Learned Adaptive Curve 2025 SIGMOD 5.1725247e-05
10,419 Managed Resource Scaling in Amazon EMR 2025 SIGMOD 5.1725247e-05
10,422 OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML 2025 SIGMOD 5.1725247e-05
10,501 Intra-Query Runtime Elasticity for Cloud-Native Data Analysis 2025 SIGMOD 5.1725247e-05
10,560 Avoiding Materialisation for Guarded Aggregate Queries 2025 VLDB 5.1725247e-05
10,577 QOVIS: Understanding and Diagnosing Query Optimizer via a Visualization-assisted Approach 2025 VLDB 5.1725247e-05
10,599 Accio: Bolt-on Query Federation 2025 VLDB 5.1725247e-05
10,670 ArrayMorph: Optimizing Hyperslab Queries on the Cloud for Machine Learning Pipelines 2025 VLDB 5.1725247e-05
10,721 Towards Designing Future-Proof Data Processing Systems 2025 VLDB 5.1725247e-05
10,743 TreeCat: Standalone Catalog Engine for Large Data Systems 2025 VLDB 5.1725247e-05
10,776 cedar: Optimized and Unified Machine Learning Input Data Pipelines 2025 VLDB 5.1725247e-05
10,784 GRewriter: Practical Query Rewriting with Automatic Rule Set Expansion in GaussDB 2025 VLDB 5.1725247e-05
10,856 CloudGlide: Deconstructing the Landscape of Cloud-Based Analytics 2025 VLDB 5.1725247e-05
10,872 LEAP: A Low-cost Spark SQL Query Optimizer using Pairwise Comparison 2025 VLDB 5.1725247e-05
10,887 IcedTea: Efficient and Responsive Time-Travel Debugging in Dataflow Systems 2025 VLDB 5.1725247e-05
10,972 Query Compilation Without Regrets 2024 SIGMOD 5.1725247e-05
11,059 Agile-Ant: Self-managing Distributed Cache Management for Cost Optimization of Big Data Applications 2024 VLDB 5.1725247e-05
11,069 OLAP on Modern Chiplet-Based Processors 2024 VLDB 5.1725247e-05
11,085 Large-Scale Metric Computation in Online Controlled Experiment Platform 2024 VLDB 5.1725247e-05
11,149 Raising the Level of Abstraction for Time-State Analytics With the Timeline Framework 2023 CIDR 5.1725247e-05
11,157 Templating Shuffles 2023 CIDR 5.1725247e-05
11,191 dsJSON: A Distributed SQL JSON Processor 2023 SIGMOD 5.1725247e-05
11,223 TEE-based General-purpose Computational Backend for Secure Delegated Data Processing 2023 SIGMOD 5.1725247e-05
11,269 Anser: Adaptive Information Sharing Framework of AnalyticDB 2023 VLDB 5.1725247e-05
11,297 XDB in Action: Decentralized Cross-Database Query Processing for Black-Box DBMSes 2023 VLDB 5.1725247e-05
11,343 Juggler: Autonomous Cost Optimization and Performance Prediction of Big Data Applications 2022 SIGMOD 5.1725247e-05
11,483 Vertex-centric Parallel Computation of SQL Queries 2021 SIGMOD 5.1725247e-05
11,489 Real-time Data Infrastructure at Uber 2021 SIGMOD 5.1725247e-05
11,526 AnyOLAP: Analytical Processing of Arbitrary Data-Intensive Applications without ETL 2021 VLDB 5.1725247e-05
11,535 Fangorn: Adaptive Execution Framework for Heterogeneous Workloads on Shared Clusters 2021 VLDB 5.1725247e-05
11,580 RASQL: A Powerful Language and its System for Big Data Applications 2020 SIGMOD 5.1725247e-05
11,677 Block as a Value for SQL over NoSQL 2019 VLDB 5.1725247e-05
11,695 Integration of Large-Scale Data Processing Systems and Traditional Parallel Database Technology 2019 VLDB 5.1725247e-05
Previous Page 4 / 5 Next

Outgoing Citations (Sorted by Pagerank)

Showing 15 of 15 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
6 Pig Latin: A Not-So-Foreign Language for Data Processing 2008 SIGMOD 0.0010826673
21 Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud 2012 VLDB 0.00057138114
42 A Comparison of Approaches to Large-Scale Data Analysis 2009 SIGMOD 0.00046908254
51 Dremel: Interactive Analysis of Web-Scale Datasets 2010 VLDB 0.00043359268
122 The EXODUS Optimizer Generator 1987 SIGMOD 0.00031727028
153 MAD Skills: New Analysis Practices for Big Data 2009 VLDB 0.00029010702
322 Impala: A Modern, Open-Source SQL Engine for Hadoop 2015 CIDR 0.00021327735
416 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018911753
520 Building Efficient Query Engines in a High-Level Language 2014 VLDB 0.00017232643
1,235 Extracting Schema from Semistructured Data 1998 SIGMOD 0.00011628464
1,717 Sinew: A SQL System for Multi-Structured Data 2014 SIGMOD 0.00010019294
1,775 Distributed Data-Parallel Computing Using a High-Level Programming Language 2009 SIGMOD 9.8630311e-05
1,825 G-OLA: Generalized On-Line Aggregation for Interactive Analysis on Big Data 2015 SIGMOD 9.7684819e-05
2,755 Inferring XML Schema Definitions from XML Data 2007 VLDB 8.2279205e-05
3,508 Rethinking Data-Intensive Science Using Scalable Analytics Systems 2015 SIGMOD 7.4266293e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
3,358 Scaling Spark in the Real World: Performance and Usability 2015 VLDB 7.5512783e-05
416 Shark: SQL and Rich Analytics at Scale 2013 SIGMOD 0.00018911753
403 SystemML: Declarative Machine Learning on Spark 2016 VLDB 0.00019232796
2,553 Big Data Analytics with Datalog Queries on Spark 2016 SIGMOD 8.4879606e-05
11,580 RASQL: A Powerful Language and its System for Big Data Applications 2020 SIGMOD 5.1725247e-05
1,139 Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark 2018 SIGMOD 0.0001209681
6,785 SparkR: Scaling R Programs with Spark 2016 SIGMOD 5.8328772e-05
9,678 Introduction to Spark 2.0 for Database Researchers 2016 SIGMOD 5.293328e-05
9,031 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB 5.3924717e-05
2,883 RaSQL: Greater Power and Performance for Big Data Analytics with Recursive-aggregate-SQL on Spark 2019 SIGMOD 8.0671021e-05