Database Paper Browser

Back to papers

Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark

Summary: Declarative Spark Structured Streaming; incrementalizes SQL/DataFrame queries, not user-built DAG. End-to-end real-time apps unifying streaming with batch analytics; code generation yields high performance; rollbacks and mixed execution in production. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5498
Venue
SIGMOD
Year
2018
Pagerank
0.00011418993
Overall Rank
1,546 | 89.26%
DOI
10.1145/3183713.3190664

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 35 of 35 citing papers.

Rank Citing Paper Year Venue Pagerank
739 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017365933
4,051 Megaphone: Latency-conscious state migration for distributed streaming dataflows 2019 VLDB 6.4932782e-05
5,131 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD 5.6704313e-05
5,357 LightSaber: Efficient Window Aggregation on Multi-core Processors 2020 SIGMOD 5.5512343e-05
5,741 Babelfish: Efficient Execution of Polyglot Queries 2022 VLDB 5.3450701e-05
5,942 Clonos: Consistent Causal Recovery for Highly-Available Streaming Dataflows 2021 SIGMOD 5.2591128e-05
6,241 Helios: Hyperscale Indexing for the Cloud & Edge 2020 VLDB 5.1357476e-05
6,430 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0587634e-05
6,726 Beyond Analytics: The Evolution of Stream Processing Systems 2020 SIGMOD 4.9444528e-05
6,760 AStream: Ad-hoc Shared Stream Processing 2019 SIGMOD 4.9304882e-05
7,371 Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile 2021 VLDB 4.744863e-05
7,941 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR 4.6089395e-05
8,078 AJoin: Ad-hoc Stream Joins at Scale 2020 VLDB 4.5873626e-05
8,214 Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines 2020 SIGMOD 4.5524768e-05
8,478 Optimization of Threshold Functions over Streams 2021 VLDB 4.4968401e-05
8,594 Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems 2020 SIGMOD 4.4844961e-05
8,785 FishStore: Faster Ingestion with Subset Hashing 2019 SIGMOD 4.4467731e-05
8,911 What's the Difference? Incremental Processing with Change Queries in Snowflake 2023 SIGMOD 4.4229886e-05
9,497 Scabbard: Single-Node Fault-Tolerant Stream Processing 2022 VLDB 4.3300131e-05
9,517 [Demo] Low-latency Spark Queries on Updatable Data 2019 SIGMOD 4.3294347e-05
9,604 GeaFlow: A Graph Extended and Accelerated Dataflow System 2023 SIGMOD 4.3136057e-05
9,806 Railgun: managing large streaming windows under MAD requirements 2021 VLDB 4.2766788e-05
9,882 Towards Resource-adaptive Query Execution in Cloud Native Databases 2024 CIDR 4.2594931e-05
9,912 Chukonu: A Fully-Featured High-Performance Big Data Framework that Integrates a Native Compute Engine into Spark 2022 VLDB 4.2524493e-05
10,077 Enjima: A Resource-Adaptive Stream Processing System 2026 SIGMOD 4.1905499e-05
10,259 Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink 2026 VLDB 4.1905499e-05
10,263 APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution 2026 VLDB 4.1905499e-05
10,292 Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things 2026 VLDB 4.1905499e-05
10,427 Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables 2025 SIGMOD 4.1905499e-05
10,518 Styx: Transactional Stateful Functions on Streaming Dataflows 2025 SIGMOD 4.1905499e-05
10,586 Agamotto: Scheduling of Deadline-Oriented Incremental Query Execution under Uncertain Resource Price 2025 VLDB 4.1905499e-05
10,743 TreeCat: Standalone Catalog Engine for Large Data Systems 2025 VLDB 4.1905499e-05
10,965 Fault Tolerance Placement in the Internet of Things 2024 SIGMOD 4.1905499e-05
11,245 Fries: Fast and Consistent Runtime Reconfiguration in Dataflow Systems with Transactional Guarantees 2023 VLDB 4.1905499e-05
11,509 Approximating Median Absolute Deviation with Bounded Error 2021 VLDB 4.1905499e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
55 Efficiently Updating Materialized Views 1986 SIGMOD 0.00065687017
66 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00061707583
191 The Design of the Borealis Stream Processing Engine 2005 CIDR 0.00035714897
402 View Maintenance in a Warehousing Environment 1995 SIGMOD 0.00024179174
524 Differential dataflow 2013 CIDR 0.00021093133
536 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020651621
590 TelegraphCQ: Continuous Dataflow Processing 2003 SIGMOD 0.00019552343
1,097 Trill: A High-Performance Incremental Query Processor for Diverse Analytics 2015 VLDB 0.00014083973
1,311 Consistency Analysis in Bloom: a CALM and Collected Approach 2011 CIDR 0.00012650881
2,204 Continuous Analytics Over Discontinuous Streams 2010 SIGMOD 9.2986554e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
2,179 Spinning Fast Iterative Data Flows 2012 VLDB 9.3632007e-05
6,760 AStream: Ad-hoc Shared Stream Processing 2019 SIGMOD 4.9304882e-05
4,925 Shared Arrangements: practical inter-query sharing for streaming dataflows 2020 VLDB 5.818597e-05
6,769 Watermarks in Stream Processing Systems: Semantics and Comparative Analysis of Apache Flink and Google Cloud Dataflow 2021 VLDB 4.9274854e-05
9,122 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB 4.3877539e-05
557 SystemML: Declarative Machine Learning on Spark 2016 VLDB 0.00020186115
7,941 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR 4.6089395e-05
3,536 Scaling Spark in the Real World: Performance and Usability 2015 VLDB 6.9938207e-05
11,734 Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams 2018 VLDB 4.1905499e-05
5,131 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD 5.6704313e-05