DBScholar

Back to papers

Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark

Summary: Declarative Spark Structured Streaming; incrementalizes SQL/DataFrame queries, not user-built DAG. End-to-end real-time apps unifying streaming with batch analytics; code generation yields high performance; rollbacks and mixed execution in production. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
h555c736ba9391777
Venue
SIGMOD
Year
2018
Pagerank
0.0001193233
Overall Rank
1,123 | 92.46%
DOI
10.1145/3183713.3190664

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{armbrust_sigmod18,
        title = {{Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark}},
        author = {Armbrust, Michael and Das, Tathagata and Torres, Joseph and Yavuz, Burak and Zhu, Shixiong and Xin, Reynold and Ghodsi, Ali and Stoica, Ion and Zaharia, Matei},
        series = {{SIGMOD} '18},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3183713.3190664},
        url = {https://dl.acm.org/doi/10.1145/3183713.3190664},
        year = {2018}
}

Incoming Citations (Sorted by Pagerank)

Showing 39 of 39 citing papers.

Rank Citing Paper Year Venue Pagerank
459 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017856221
3,398 Megaphone: Latency-conscious state migration for distributed streaming dataflows 2019 VLDB 7.3353479e-05
4,866 Clonos: Consistent Causal Recovery for Highly-Available Streaming Dataflows 2021 SIGMOD 6.3754084e-05
4,884 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD 6.3710319e-05
4,918 LightSaber: Efficient Window Aggregation on Multi-core Processors 2020 SIGMOD 6.3548575e-05
5,230 Babelfish: Efficient Execution of Polyglot Queries 2022 VLDB 6.2189001e-05
6,174 Helios: Hyperscale Indexing for the Cloud & Edge 2020 VLDB 5.8612829e-05
6,225 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.8455305e-05
6,266 Beyond Analytics: The Evolution of Stream Processing Systems 2020 SIGMOD 5.8303329e-05
6,599 AStream: Ad-hoc Shared Stream Processing 2019 SIGMOD 5.7404438e-05
6,787 Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile 2021 VLDB 5.6820544e-05
7,267 What's the Difference? Incremental Processing with Change Queries in Snowflake 2023 SIGMOD 5.5698185e-05
8,014 AJoin: Ad-hoc Stream Joins at Scale 2020 VLDB 5.4070928e-05
8,404 Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines 2020 SIGMOD 5.3385814e-05
8,424 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR 5.3350162e-05
8,678 Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems 2020 SIGMOD 5.2907845e-05
8,864 Optimization of Threshold Functions over Streams 2021 VLDB 5.262266e-05
8,953 FishStore: Faster Ingestion with Subset Hashing 2019 SIGMOD 5.2515834e-05
9,226 Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables 2025 SIGMOD 5.2056825e-05
9,810 Scabbard: Single-Node Fault-Tolerant Stream Processing 2022 VLDB 5.1257999e-05
9,832 [Demo] Low-latency Spark Queries on Updatable Data 2019 SIGMOD 5.1245795e-05
9,916 GeaFlow: A Graph Extended and Accelerated Dataflow System 2023 SIGMOD 5.1103839e-05
10,158 Railgun: managing large streaming windows under MAD requirements 2021 VLDB 5.0706053e-05
10,222 Towards Resource-adaptive Query Execution in Cloud Native Databases 2024 CIDR 5.0572221e-05
10,227 TreeCat: Standalone Catalog Engine for Large Data Systems 2025 VLDB 5.0571508e-05
10,280 Chukonu: A Fully-Featured High-Performance Big Data Framework that Integrates a Native Compute Engine into Spark 2022 VLDB 5.0455234e-05
10,566 Enjima: A Resource-Adaptive Stream Processing System 2026 SIGMOD 4.9793485e-05
10,729 Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink 2026 VLDB 4.9793485e-05
10,733 APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution 2026 VLDB 4.9793485e-05
10,767 IncreQueryFusion: On-demand Data Fusion Framework in Dynamic Data Lakes 2026 VLDB 4.9793485e-05
10,846 Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things 2026 VLDB 4.9793485e-05
10,862 Fugue: Online Elasticity for Distributed Stateful Stream Processing 2026 VLDB 4.9793485e-05
10,927 A Decade of Apache Spark Structured Streaming: How We Evolved The Architecture To Meet Real-World Needs 2026 VLDB 4.9793485e-05
11,036 The Dataflow Model Revisited Or: That Feeling When You Realize Every Problem You've Been Solving Is a Database Problem 2026 VLDB 4.9793485e-05
11,199 Styx: Transactional Stateful Functions on Streaming Dataflows 2025 SIGMOD 4.9793485e-05
11,248 Agamotto: Scheduling of Deadline-Oriented Incremental Query Execution under Uncertain Resource Price 2025 VLDB 4.9793485e-05
11,521 Fault Tolerance Placement in the Internet of Things 2024 SIGMOD 4.9793485e-05
11,755 Fries: Fast and Consistent Runtime Reconfiguration in Dataflow Systems with Transactional Guarantees 2023 VLDB 4.9793485e-05
12,007 Approximating Median Absolute Deviation with Bounded Error 2021 VLDB 4.9793485e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
23 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00055406774
39 Efficiently Updating Materialized Views 1986 SIGMOD 0.00046602544
127 The Design of the Borealis Stream Processing Engine 2005 CIDR 0.00030427614
325 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020964941
398 View Maintenance in a Warehousing Environment 1995 SIGMOD 0.00019152162
442 Differential dataflow 2013 CIDR 0.00018210463
510 TelegraphCQ: Continuous Dataflow Processing 2003 SIGMOD 0.00017065714
685 Trill: A High-Performance Incremental Query Processor for Diverse Analytics 2015 VLDB 0.00014782777
1,245 Consistency Analysis in Bloom: a CALM and Collected Approach 2011 CIDR 0.00011353116
1,985 Continuous Analytics Over Discontinuous Streams 2010 SIGMOD 9.2521678e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 2,227 Spinning Fast Iterative Data Flows 2012 VLDB
2 4,400 Shared Arrangements: practical inter-query sharing for streaming dataflows 2020 VLDB
3 6,620 Watermarks in Stream Processing Systems: Semantics and Comparative Analysis of Apache Flink and Google Cloud Dataflow 2021 VLDB
4 9,321 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB
5 415 SystemML: Declarative Machine Learning on Spark 2016 VLDB
6 8,424 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR
7 3,455 Scaling Spark in the Real World: Performance and Usability 2015 VLDB
8 12,232 Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams 2018 VLDB
9 4,884 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD
10 10,927 A Decade of Apache Spark Structured Streaming: How We Evolved The Architecture To Meet Real-World Needs 2026 VLDB