DBScholar

Back to papers

Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka

Summary: Kafka's persistent log provides strong correctness under failures and out-of-order data. Kafka Streams uses read–process–write over log-append state with idempotent/transactional exactly-once and revision-based speculative processing for early results, enabling scalable correctness–performance–cost trade-offs. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
6307
Venue
SIGMOD
Year
2021
Pagerank
6.7304854e-05
Overall Rank
4,389 | 69.89%
DOI
10.1145/3448016.3457556

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@inproceedings{wang_sigmod21,
        title = {{Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka}},
        author = {Wang, Guozhang and Chen, Lei and Dikshit, Ayusman and Gustafson, Jason and Chen, Boyang and Sax, Matthias J. and Roesler, John and Blee-Goldman, Sophie and Cadonna, Bruno and Mehta, Apurva and Madan, Varun and Rao, Jun},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3457556},
        url = {https://dl.acm.org/doi/10.1145/3448016.3457556},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 7 of 7 citing papers.

Rank Citing Paper Year Venue Pagerank
5,821 StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance 2023 VLDB 6.07598e-05
6,502 Watermarks in Stream Processing Systems: Semantics and Comparative Analysis of Apache Flink and Google Cloud Dataflow 2021 VLDB 5.8604458e-05
8,375 Serverless State Management Systems 2024 CIDR 5.4391978e-05
9,064 What's the Difference? Incremental Processing with Change Queries in Snowflake 2023 SIGMOD 5.3251649e-05
9,468 Disaggregated State Management in Apache Flink 2.0 2025 VLDB 5.2634238e-05
9,895 DARQ Matter Binds Everything: Performant and Composable Cloud Programming via Resilient Steps 2023 SIGMOD 5.1997534e-05
11,287 Towards Millions of Database Transmission Services in the Cloud 2024 VLDB 5.093636e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 21 of 21 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
30 SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets 2008 VLDB 0.00051174276
111 TelegraphCQ: Continuous Dataflow Processing for an Uncertain World 2003 CIDR 0.00032990214
127 The Design of the Borealis Stream Processing Engine 2005 CIDR 0.00030738755
224 MillWheel: Fault-Tolerant Stream Processing at Internet Scale 2013 VLDB 0.00024130894
230 Gigascope: A Stream Database for Network Applications 2003 SIGMOD 0.00023891474
361 The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing 2015 VLDB 0.00020138717
424 Flexible Time Management in Data Stream Systems 2004 PODS 0.0001870735
710 Trill: A High-Performance Incremental Query Processor for Diverse Analytics 2015 VLDB 0.00014715033
983 Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management 2013 SIGMOD 0.0001283214
1,223 Out-of-Order Processing: A New Architecture for High-Performance Stream Systems 2008 VLDB 0.00011602265
1,224 Dhalion: Self-Regulating Stream Processing in Heron 2017 VLDB 0.00011596911
1,422 State Management in Apache Flink: Consistent Stateful Distributed Stream Processing 2017 VLDB 0.00010823039
1,888 Aurora: A Data Stream Management System 2003 SIGMOD 9.5338565e-05
1,895 Samza: Stateful Scalable Stream Processing at LinkedIn 2017 VLDB 9.5260291e-05
1,915 S-Store: Streaming Meets Transaction Processing 2015 VLDB 9.4884706e-05
3,117 Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems 2018 VLDB 7.7382559e-05
3,531 Megaphone: Latency-conscious state migration for distributed streaming dataflows 2019 VLDB 7.337549e-05
4,278 Consistent Regions: Guaranteed Tuple Processing in IBM Streams 2016 VLDB 6.7871785e-05
5,419 Building a Replicated Logging System with Apache Kafka 2015 VLDB 6.2255013e-05
5,719 Optimal and General Out-of-Order Sliding-Window Aggregation 2019 VLDB 6.1094861e-05
6,734 Liquid: Unifying Nearline and Offline Big Data Integration 2015 CIDR 5.7879923e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 6,502 Watermarks in Stream Processing Systems: Semantics and Comparative Analysis of Apache Flink and Google Cloud Dataflow 2021 VLDB
2 9,562 Kora: A Cloud-Native Event Streaming Platform For Kafka 2023 VLDB
3 8,258 Correctness in Stream Processing: Challenges and Opportunities 2022 CIDR
4 1,895 Samza: Stateful Scalable Stream Processing at LinkedIn 2017 VLDB
5 12,022 Toward High-Performance Distributed Stream Processing via Approximate Fault Tolerance 2017 VLDB
6 1,789 Fault-Tolerance in the Borealis Distributed Stream Processing System 2005 SIGMOD
7 11,558 KafkaDirect: Zero-copy Data Access for Apache Kafka over RDMA Networks 2022 SIGMOD
8 1,422 State Management in Apache Flink: Consistent Stateful Distributed Stream Processing 2017 VLDB
9 11,086 How Reliable Are Streams? End-to-End Processing-Guarantee Validation and Performance Benchmarking of Stream Processing Systems 2025 VLDB
10 5,419 Building a Replicated Logging System with Apache Kafka 2015 VLDB