Database Paper Browser

Back to papers

Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark

Summary: Declarative Spark Structured Streaming; incrementalizes SQL/DataFrame queries, not user-built DAG. End-to-end real-time apps unifying streaming with batch analytics; code generation yields high performance; rollbacks and mixed execution in production. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
5497
Venue
SIGMOD
Year
2018
Pagerank
0.00011431383
Overall Rank
1,548 | 89.24%
DOI
10.1145/3183713.3190664

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 35 of 35 citing papers.

Rank Citing Paper Year Venue Pagerank
746 Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores 2020 VLDB 0.00017326979
4,044 Megaphone: Latency-conscious state migration for distributed streaming dataflows 2019 VLDB 6.4995312e-05
5,130 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD 5.6755067e-05
5,193 LightSaber: Efficient Window Aggregation on Multi-core Processors 2020 SIGMOD 5.6371049e-05
5,731 Babelfish: Efficient Execution of Polyglot Queries 2022 VLDB 5.3502065e-05
5,939 Clonos: Consistent Causal Recovery for Highly-Available Streaming Dataflows 2021 SIGMOD 5.2641681e-05
6,242 Helios: Hyperscale Indexing for the Cloud & Edge 2020 VLDB 5.1408379e-05
6,436 Providing Streaming Joins as a Service at Facebook 2018 VLDB 5.0636254e-05
6,721 Beyond Analytics: The Evolution of Stream Processing Systems 2020 SIGMOD 4.9492015e-05
6,759 AStream: Ad-hoc Shared Stream Processing 2019 SIGMOD 4.9352213e-05
7,373 Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile 2021 VLDB 4.7494183e-05
7,938 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR 4.613363e-05
8,075 AJoin: Ad-hoc Stream Joins at Scale 2020 VLDB 4.5917655e-05
8,217 Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines 2020 SIGMOD 4.5568298e-05
8,480 Optimization of Threshold Functions over Streams 2021 VLDB 4.5011552e-05
8,596 Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems 2020 SIGMOD 4.4887993e-05
8,788 FishStore: Faster Ingestion with Subset Hashing 2019 SIGMOD 4.451039e-05
8,909 What's the Difference? Incremental Processing with Change Queries in Snowflake 2023 SIGMOD 4.427232e-05
9,496 Scabbard: Single-Node Fault-Tolerant Stream Processing 2022 VLDB 4.3341665e-05
9,516 [Demo] Low-latency Spark Queries on Updatable Data 2019 SIGMOD 4.3335877e-05
9,604 GeaFlow: A Graph Extended and Accelerated Dataflow System 2023 SIGMOD 4.3177432e-05
9,803 Railgun: managing large streaming windows under MAD requirements 2021 VLDB 4.2807806e-05
9,883 Towards Resource-adaptive Query Execution in Cloud Native Databases 2024 CIDR 4.2635782e-05
9,913 Chukonu: A Fully-Featured High-Performance Big Data Framework that Integrates a Native Compute Engine into Spark 2022 VLDB 4.2565279e-05
10,077 Enjima: A Resource-Adaptive Stream Processing System 2026 SIGMOD 4.1945683e-05
10,259 Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink 2026 VLDB 4.1945683e-05
10,263 APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution 2026 VLDB 4.1945683e-05
10,280 Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things 2026 VLDB 4.1945683e-05
10,417 Streaming Democratized: Ease Across the Latency Spectrum with Delayed View Semantics and Snowflake Dynamic Tables 2025 SIGMOD 4.1945683e-05
10,509 Styx: Transactional Stateful Functions on Streaming Dataflows 2025 SIGMOD 4.1945683e-05
10,577 Agamotto: Scheduling of Deadline-Oriented Incremental Query Execution under Uncertain Resource Price 2025 VLDB 4.1945683e-05
10,736 TreeCat: Standalone Catalog Engine for Large Data Systems 2025 VLDB 4.1945683e-05
10,962 Fault Tolerance Placement in the Internet of Things 2024 SIGMOD 4.1945683e-05
11,243 Fries: Fast and Consistent Runtime Reconfiguration in Dataflow Systems with Transactional Guarantees 2023 VLDB 4.1945683e-05
11,505 Approximating Median Absolute Deviation with Bounded Error 2021 VLDB 4.1945683e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 10 of 10 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
55 Efficiently Updating Materialized Views 1986 SIGMOD 0.00065762967
66 Spark SQL: Relational Data Processing in Spark 2015 SIGMOD 0.00061639801
191 The Design of the Borealis Stream Processing Engine 2005 CIDR 0.00035738595
401 View Maintenance in a Warehousing Environment 1995 SIGMOD 0.00024214488
522 Differential dataflow 2013 CIDR 0.00021099241
538 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020678804
591 TelegraphCQ: Continuous Dataflow Processing 2003 SIGMOD 0.00019569071
1,098 Trill: A High-Performance Incremental Query Processor for Diverse Analytics 2015 VLDB 0.00014114442
1,310 Consistency Analysis in Bloom: a CALM and Collected Approach 2011 CIDR 0.00012658593
2,198 Continuous Analytics Over Discontinuous Streams 2010 SIGMOD 9.308495e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
6,759 AStream: Ad-hoc Shared Stream Processing 2019 SIGMOD 4.9352213e-05
2,172 Spinning Fast Iterative Data Flows 2012 VLDB 9.3706587e-05
4,920 Shared Arrangements: practical inter-query sharing for streaming dataflows 2020 VLDB 5.8241888e-05
6,767 Watermarks in Stream Processing Systems: Semantics and Comparative Analysis of Apache Flink and Google Cloud Dataflow 2021 VLDB 4.9322174e-05
9,124 Dynamic Speculative Optimizations for SQL Compilation in Apache Spark 2020 VLDB 4.391961e-05
557 SystemML: Declarative Machine Learning on Spark 2016 VLDB 0.00020197988
7,938 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR 4.613363e-05
3,535 Scaling Spark in the Real World: Performance and Usability 2015 VLDB 6.9992495e-05
11,728 Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams 2018 VLDB 4.1945683e-05
5,130 One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables 2019 SIGMOD 5.6755067e-05