Dhalion: Self-Regulating Stream Processing in Heron

Summary: Proposes Dhalion, a self-regulating layer for streaming systems on Heron that autonomously tunes knobs to meet throughput SLOs under variable load. Policy-driven reconfiguration scales resources and topology to sustain QoS, with cloud evaluation and open-source policy plans. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h8e37ccdc07b7f69c
Venue: VLDB
Year: 2017
Pagerank: 0.00011363001
Overall Rank: 1,244 | 91.67%
DOI: 10.14778/3137765.3137785
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Avrilia Floratou (Microsoft)
2. Ashvin Agrawal (Microsoft)
3. Bill Graham (Twitter)
4. Sriram Rao (Microsoft)
5. Karthik Ramasamy (Streamlio)

BibTeX Citation

@article{floratou_vldb17,
        title = {{Dhalion: Self-Regulating Stream Processing in Heron}},
        author = {Floratou, Avrilia and Agrawal, Ashvin and Graham, Bill and Rao, Sriram and Ramasamy, Karthik},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {12},
        pages = {1825--1836},
        doi = {10.14778/3137765.3137785},
        url = {https://doi.org/10.14778/3137765.3137785},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 27 of 27 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,004	Moneyball: Proactive Auto-Scaling in Microsoft Azure SQL Database Serverless	2022	VLDB	9.1854436e-05
3,043	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB	7.7045781e-05
3,409	Megaphone: Latency-conscious state migration for distributed streaming dataflows	2019	VLDB	7.3076317e-05
3,691	Cloudy with High Chance of DBMS: A 10-year Prediction for Enterprise-Grade ML	2020	CIDR	7.0771037e-05
4,129	Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines	2020	SIGMOD	6.7751787e-05
4,475	Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka	2021	SIGMOD	6.5699934e-05
4,633	Efficient Placement of Decomposable Aggregation Functions for Stream Processing over Large Geo-Distributed Topologies	2024	VLDB	6.4771063e-05
5,952	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB	5.9182633e-05
6,281	Beyond Analytics: The Evolution of Stream Processing Systems	2020	SIGMOD	5.8083033e-05
6,921	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.63068e-05
7,432	POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least Resistance	2024	VLDB	5.5133928e-05
7,830	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.4294933e-05
8,372	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3287108e-05
8,427	Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines	2020	SIGMOD	5.3184107e-05
9,511	Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing	2019	SIGMOD	5.1516241e-05
9,854	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB	5.1064323e-05
9,998	Lorentz: Learned SKU Recommendation Using Profile Data (DMDS)	2024	SIGMOD	5.0845476e-05
10,123	ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems	2023	VLDB	5.0638787e-05
10,617	Accelerating Stream Processing Engines via Hardware Offloading	2026	SIGMOD	4.9605343e-05
10,800	Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink	2026	VLDB	4.9605343e-05
10,804	APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution	2026	VLDB	4.9605343e-05
10,916	Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things	2026	VLDB	4.9605343e-05
11,267	Styx: Transactional Stateful Functions on Streaming Dataflows	2025	SIGMOD	4.9605343e-05
11,276	Scalable Complex Event Processing on Video Streams	2025	SIGMOD	4.9605343e-05
11,614	Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service	2024	VLDB	4.9605343e-05
12,239	Minimizing Cost by Reducing Scaling Operations in Distributed Stream Processing	2019	VLDB	4.9605343e-05
12,281	Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams	2018	VLDB	4.9605343e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
128	The Design of the Borealis Stream Processing Engine	2005	CIDR	0.00030317705
219	MillWheel: Fault-Tolerant Stream Processing at Internet Scale	2013	VLDB	0.0002429959
224	Self-Driving Database Management Systems	2017	CIDR	0.00023973185
233	Storm @Twitter	2014	SIGMOD	0.00023754096
607	Twitter Heron: Stream Processing at Scale	2015	SIGMOD	0.00015577054
893	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00013199466
944	Starfish: A Self-tuning System for Big Data Analytics	2011	CIDR	0.00012927479

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	233	Storm @Twitter	2014	SIGMOD
2	9,668	Shrink - Prescribing Resiliency Solutions for Streaming	2017	VLDB
3	7,333	Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning	2018	VLDB
4	3,043	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB
5	11,197	Oceanus: Enable SLO-Aware Vertical Autoscaling for Cloud-Native Streaming Services in Tencent	2025	SIGMOD
6	10,197	Railgun: managing large streaming windows under MAD requirements	2021	VLDB
7	5,952	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB
8	9,232	Dalton: Learned Partitioning for Distributed Data Streams	2023	VLDB
9	607	Twitter Heron: Stream Processing at Scale	2015	SIGMOD
10	9,854	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB