Darwin: Scale-In Stream Processing

Summary: Propose “scale-in”: compile-tailored stream processing that maximizes utilization of existing single- and multi-node hardware and supports recoverable larger-than-memory state, avoiding the scale-up vs availability tradeoff. Darwin prototype implements this and reports ~10× speedups vs scale-out systems while matching scale-up throughput. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 449
Venue: CIDR
Year: 2022
Pagerank: 5.1633991e-05
Overall Rank: 10,070 | 30.92%
DOI: -

Incoming Non-self Citations Over Time

Authors

1. Lawrence Benson (Hasso Plattner Institute; University of Potsdam)
2. Tilmann Rabl (Hasso Plattner Institute; University of Potsdam)

BibTeX Citation

@inproceedings{benson_cidr22,
        address = {Amsterdam, Netherlands},
        series = {{CIDR} '22},
        title = {{Darwin: Scale-In Stream Processing}},
        booktitle = {Proceedings of the {Conference} on {Innovative} {Data} {Systems} {Research}},
        author = {Benson, Lawrence and Rabl, Tilmann},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
8,443	Analyzing Vectorized Hash Tables Across CPU Architectures	2023	VLDB	5.4243766e-05
10,072	Query Compilation Without Regrets	2024	SIGMOD	5.1624689e-05

Outgoing Citations (Sorted by Pagerank)

Showing 13 of 13 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
23	Efficiently Compiling Efficient Query Plans for Modern Hardware	2011	VLDB	0.00054886415
614	Faster: A Concurrent Key-Value Store with In-Place Updates	2018	SIGMOD	0.00015720975
710	Trill: A High-Performance Incremental Query Processor for Diverse Analytics	2015	VLDB	0.00014715033
2,471	Viper: An Efficient Hybrid PMem-DRAM Key-Value Store	2021	VLDB	8.5329393e-05
3,060	SABER: Window-Based Hybrid Stream Processing for Heterogeneous Architectures	2016	SIGMOD	7.8062155e-05
3,647	Maximizing Persistent Memory Bandwidth Utilization for OLAP Workloads	2021	SIGMOD	7.2271935e-05
3,851	Analyzing Efficient Stream Processing on Modern Hardware	2019	VLDB	7.0735328e-05
3,868	DFI: The Data Flow Interface for High-Speed Networks	2021	SIGMOD	7.0614899e-05
4,888	Enabling Low Tail Latency on Multicore Key-Value Stores	2020	VLDB	6.4609843e-05
5,206	BriskStream: Scaling Data Stream Processing on Shared-Memory Multicore Architectures	2019	SIGMOD	6.3180444e-05
5,765	Charting the Design Space of Query Execution using VOILA	2021	VLDB	6.0953705e-05
6,349	Grizzly: Efficient Stream Processing Through Adaptive Query Compilation	2020	SIGMOD	5.9049304e-05
7,294	Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile	2021	VLDB	5.6532162e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,190	Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark	2018	SIGMOD
2	4,918	Massive Scale-out of Expensive Continuous Queries	2011	VLDB
3	6,349	Grizzly: Efficient Stream Processing Through Adaptive Query Compilation	2020	SIGMOD
4	8,663	Scalable Delivery of Stream Query Result	2009	VLDB
5	10,337	Accelerating Stream Processing Engines via Hardware Offloading	2026	SIGMOD
6	2,156	Quickstep: A Data Platform Based on the Scaling-Up Approach	2018	VLDB
7	11,665	Klink: Progress-Aware Scheduling for Streaming Data Systems	2021	SIGMOD
8	4,279	Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines	2020	SIGMOD
9	983	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD
10	3,851	Analyzing Efficient Stream Processing on Modern Hardware	2019	VLDB