Dhalion: Self-Regulating Stream Processing in Heron

Summary: Proposes Dhalion, a self-regulating layer for streaming systems on Heron that autonomously tunes knobs to meet throughput SLOs under variable load. Policy-driven reconfiguration scales resources and topology to sustain QoS, with cloud evaluation and open-source policy plans. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 11652
Venue: VLDB
Year: 2017
Pagerank: 0.00011596911
Overall Rank: 1,224 | 91.61%
DOI: 10.14778/3137765.3137785

Incoming Non-self Citations Over Time

Authors

1. Avrilia Floratou (Microsoft)
2. Ashvin Agrawal (Microsoft)
3. Bill Graham (Twitter)
4. Sriram Rao (Microsoft)
5. Karthik Ramasamy (Streamlio)

BibTeX Citation

@article{floratou_vldb17,
        title = {{Dhalion: Self-Regulating Stream Processing in Heron}},
        author = {Floratou, Avrilia and Agrawal, Ashvin and Graham, Bill and Rao, Sriram and Ramasamy, Karthik},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {12},
        pages = {1825--1836},
        doi = {10.14778/3137765.3137785},
        url = {https://doi.org/10.14778/3137765.3137785},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 27 of 27 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,226	Moneyball: Proactive Auto-Scaling in Microsoft Azure SQL Database Serverless	2022	VLDB	8.9174082e-05
3,117	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB	7.7382559e-05
3,531	Megaphone: Latency-conscious state migration for distributed streaming dataflows	2019	VLDB	7.337549e-05
3,614	Cloudy with High Chance of DBMS: A 10-year Prediction for Enterprise-Grade ML	2020	CIDR	7.2568185e-05
4,279	Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines	2020	SIGMOD	6.7865878e-05
4,389	Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka	2021	SIGMOD	6.7304854e-05
4,513	Efficient Placement of Decomposable Aggregation Functions for Stream Processing over Large Geo-Distributed Topologies	2024	VLDB	6.6509008e-05
5,821	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB	6.07598e-05
6,680	Beyond Analytics: The Evolution of Stream Processing Systems	2020	SIGMOD	5.8049764e-05
6,781	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.7764885e-05
7,661	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.5736026e-05
8,193	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.4696038e-05
8,309	Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines	2020	SIGMOD	5.4560857e-05
8,494	POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least Resistance	2024	VLDB	5.4142129e-05
9,336	Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing	2019	SIGMOD	5.2878276e-05
9,637	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB	5.2434488e-05
9,898	ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems	2023	VLDB	5.1997534e-05
10,337	Accelerating Stream Processing Engines via Hardware Offloading	2026	SIGMOD	5.093636e-05
10,547	Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink	2026	VLDB	5.093636e-05
10,551	APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution	2026	VLDB	5.093636e-05
10,579	Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things	2026	VLDB	5.093636e-05
10,782	Styx: Transactional Stateful Functions on Streaming Dataflows	2025	SIGMOD	5.093636e-05
10,795	Scalable Complex Event Processing on Video Streams	2025	SIGMOD	5.093636e-05
11,180	Lorentz: Learned SKU Recommendation Using Profile Data (DMDS)	2024	SIGMOD	5.093636e-05
11,222	Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service	2024	VLDB	5.093636e-05
11,890	Minimizing Cost by Reducing Scaling Operations in Distributed Stream Processing	2019	VLDB	5.093636e-05
11,933	Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams	2018	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
127	The Design of the Borealis Stream Processing Engine	2005	CIDR	0.00030738755
220	Storm @Twitter	2014	SIGMOD	0.00024244587
224	MillWheel: Fault-Tolerant Stream Processing at Internet Scale	2013	VLDB	0.00024130894
234	Self-Driving Database Management Systems	2017	CIDR	0.00023810722
612	Twitter Heron: Stream Processing at Scale	2015	SIGMOD	0.0001573018
895	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00013357681
923	Starfish: A Self-tuning System for Big Data Analytics	2011	CIDR	0.00013189886

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	220	Storm @Twitter	2014	SIGMOD
2	9,531	Shrink - Prescribing Resiliency Solutions for Streaming	2017	VLDB
3	7,194	Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning	2018	VLDB
4	3,117	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB
5	10,695	Oceanus: Enable SLO-Aware Vertical Autoscaling for Cloud-Native Streaming Services in Tencent	2025	SIGMOD
6	9,968	Railgun: managing large streaming windows under MAD requirements	2021	VLDB
7	5,821	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB
8	9,128	Dalton: Learned Partitioning for Distributed Data Streams	2023	VLDB
9	612	Twitter Heron: Stream Processing at Scale	2015	SIGMOD
10	9,637	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB