A Decade of Apache Spark Structured Streaming: How We Evolved The Architecture To Meet Real-World Needs

Summary: A decade of productionizing Spark Structured Streaming’s microbatch architecture for enterprise workloads. Microbatch pipelining, source decoupling, flexible transformWithState, observability, and sandboxed access control address utilization, heterogeneous sources, stateful computation, and security. (summarized by gpt-5.6-luna on Aug 28 2026)

Paper ID: hfec86e8a082c3583
Venue: VLDB
Year: 2026
Pagerank: 4.9793485e-05
Overall Rank: 10,927 | 26.54%
DOI: 10.14778/3827998.3828023

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

1. Anish Shrigondekar (Databricks)
2. Boyang Jerry Peng (Databricks)
3. Siying Dong (Databricks)
4. Jungtaek Lim (Databricks)
5. Huanli Wang (Databricks)
6. Bo Gao (Databricks)
7. Eric Marnadi (Databricks)
8. Jason Teoh (Databricks)
9. B. Micheal Okutubo (Databricks)
10. Livia Zhu (Databricks)
11. Yuchen Liu (Databricks)
12. Chloe Xia (Databricks)
13. Abhay Bothra (Databricks)
14. Sagar Mittal (Databricks)
15. Karthik Ramasamy (Databricks)
16. Michael Armbrust (Databricks)
17. Fatih Emekci (Databricks)
18. Shan Shan Huang (Databricks)
19. Shrikanth Shankar (Databricks)
20. Indrajit Roy (Databricks)

BibTeX Citation

@article{shrigondekar_vldb26,
        title = {{A Decade of Apache Spark Structured Streaming: How We Evolved The Architecture To Meet Real-World Needs}},
        author = {Shrigondekar, Anish and Peng, Boyang Jerry and Dong, Siying and Lim, Jungtaek and Wang, Huanli and Gao, Bo and Marnadi, Eric and Teoh, Jason and Okutubo, B. Micheal and Zhu, Livia and Liu, Yuchen and Xia, Chloe and Bothra, Abhay and Mittal, Sagar and Ramasamy, Karthik and Armbrust, Michael and Emekci, Fatih and Huang, Shan Shan and Shankar, Shrikanth and Roy, Indrajit},
        journal = {PVLDB},
        series = {{VLDB} '26},
        volume = {19},
        number = {12},
        pages = {4156--4168},
        doi = {10.14778/3827998.3828023},
        url = {https://doi.org/10.14778/3827998.3828023},
        year = {2026}
}

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
21	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00056855599
23	Spark SQL: Relational Data Processing in Spark	2015	SIGMOD	0.00055406774
218	MillWheel: Fault-Tolerant Stream Processing at Internet Scale	2013	VLDB	0.00024390324
231	Storm @Twitter	2014	SIGMOD	0.00023841089
325	The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing	2015	VLDB	0.00020964941
459	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017856221
1,123	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD	0.0001193233
1,441	Cloudburst: Stateful Functions-as-a-Service	2020	VLDB	0.00010641542
1,487	Photon: A Fast Query Engine for Lakehouse Systems	2022	SIGMOD	0.00010521722
1,771	Samza: Stateful Scalable Stream Processing at LinkedIn	2017	VLDB	9.6803752e-05
5,027	Building a Replicated Logging System with Apache Kafka	2015	VLDB	6.3088256e-05
6,920	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB	5.6441822e-05
7,582	Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond	2025	SIGMOD	5.4910839e-05
7,996	Databricks Lakeguard: Supporting Fine-grained Access Control and Multi-user Capabilities for Apache Spark Workloads	2025	SIGMOD	5.4108483e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	968	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD
2	3,922	Analyzing Efficient Stream Processing on Modern Hardware	2019	VLDB
3	6,920	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB
4	4,884	One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables	2019	SIGMOD
5	1,402	State Management in Apache Flink: Consistent Stateful Distributed Stream Processing	2017	VLDB
6	13,623	Blink Twice - Automatic Workload Pinning and Regression Detection for Versionless Apache Spark using Retries	2025	SIGMOD
7	6,266	Beyond Analytics: The Evolution of Stream Processing Systems	2020	SIGMOD
8	12,232	Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams	2018	VLDB
9	3,455	Scaling Spark in the Real World: Performance and Usability	2015	VLDB
10	1,123	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD