Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams
Summary: IBM Streams' Beam runner optimizes event-time windows by indexing inter-dependent states, garbage-collecting stale keys, and tuning bundle sizes. On NEXMark, it outruns Flink and Spark, demonstrating efficient, enterprise Beam integration on Streams. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Shen Li (International Business Machines Research AI)
- 2. Paul Gerver (International Business Machines Watson Cloud Platform)
- 3. John MacMillan (International Business Machines Watson Cloud Platform)
- 4. Daniel Debrunner (International Business Machines Watson Cloud Platform)
- 5. William Marshall (International Business Machines Watson Cloud Platform)
- 6. Kun-Lung Wu (International Business Machines Research AI)
BibTeX Citation
@article{li_vldb18,
title = {{Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams}},
author = {Li, Shen and Gerver, Paul and MacMillan, John and Debrunner, Daniel and Marshall, William and Wu, Kun-Lung},
journal = {PVLDB},
series = {{VLDB} '18},
volume = {11},
number = {12},
pages = {1742--1754},
doi = {10.14778/3229863.3229864},
url = {https://doi.org/10.14778/3229863.3229864},
year = {2018}
}Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 9 of 9 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 220 | Storm @Twitter | 2014 | SIGMOD | 0.00024244587 |
| 224 | MillWheel: Fault-Tolerant Stream Processing at Internet Scale | 2013 | VLDB | 0.00024130894 |
| 361 | The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing | 2015 | VLDB | 0.00020138717 |
| 612 | Twitter Heron: Stream Processing at Scale | 2015 | SIGMOD | 0.0001573018 |
| 1,224 | Dhalion: Self-Regulating Stream Processing in Heron | 2017 | VLDB | 0.00011596911 |
| 1,422 | State Management in Apache Flink: Consistent Stateful Distributed Stream Processing | 2017 | VLDB | 0.00010823039 |
| 1,895 | Samza: Stateful Scalable Stream Processing at LinkedIn | 2017 | VLDB | 9.5260291e-05 |
| 2,802 | General Incremental Sliding-Window Aggregation | 2015 | VLDB | 8.1093063e-05 |
| 4,278 | Consistent Regions: Guaranteed Tuple Processing in IBM Streams | 2016 | VLDB | 6.7871785e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 3,532 | Scalable Distributed Stream Join Processing | 2015 | SIGMOD |
| 2 | 4,534 | Shared Arrangements: practical inter-query sharing for streaming dataflows | 2020 | VLDB |
| 3 | 8,248 | Demonstrating PDSP-Bench: A Benchmarking System for Parallel and Distributed Stream Processing | 2025 | SIGMOD |
| 4 | 6,012 | Parallel Index-based Stream Join on a Multicore CPU | 2020 | SIGMOD |
| 5 | 6,482 | AStream: Ad-hoc Shared Stream Processing | 2019 | SIGMOD |
| 6 | 12,333 | Building User-defined Runtime Adaptation Routines for Stream Processing Applications | 2012 | VLDB |
| 7 | 3,851 | Analyzing Efficient Stream Processing on Modern Hardware | 2019 | VLDB |
| 8 | 8,536 | Changing Engines in Midstream: A Java Stream Computational Model for Big Data Processing | 2014 | VLDB |
| 9 | 4,907 | One SQL to Rule Them All – an Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables | 2019 | SIGMOD |
| 10 | 1,190 | Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark | 2018 | SIGMOD |