Dhalion: Self-Regulating Stream Processing in Heron

Summary: Proposes Dhalion, a self-regulating layer for streaming systems on Heron that autonomously tunes knobs to meet throughput SLOs under variable load. Policy-driven reconfiguration scales resources and topology to sustain QoS, with cloud evaluation and open-source policy plans. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h8e37ccdc07b7f69c
Venue: VLDB
Year: 2017
Pagerank: 0.00011400483
Overall Rank: 1,236 | 91.70%
DOI: 10.14778/3137765.3137785
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Avrilia Floratou (Microsoft)
2. Ashvin Agrawal (Microsoft)
3. Bill Graham (Twitter)
4. Sriram Rao (Microsoft)
5. Karthik Ramasamy (Streamlio)

BibTeX Citation

@article{floratou_vldb17,
        title = {{Dhalion: Self-Regulating Stream Processing in Heron}},
        author = {Floratou, Avrilia and Agrawal, Ashvin and Graham, Bill and Rao, Sriram and Ramasamy, Karthik},
        journal = {PVLDB},
        series = {{VLDB} '17},
        volume = {10},
        number = {12},
        pages = {1825--1836},
        doi = {10.14778/3137765.3137785},
        url = {https://doi.org/10.14778/3137765.3137785},
        year = {2017}
}

Incoming Citations (Sorted by Pagerank)

Showing 27 of 27 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,996	Moneyball: Proactive Auto-Scaling in Microsoft Azure SQL Database Serverless	2022	VLDB	9.213369e-05
3,034	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB	7.7301387e-05
3,398	Megaphone: Latency-conscious state migration for distributed streaming dataflows	2019	VLDB	7.3318754e-05
3,685	Cloudy with High Chance of DBMS: A 10-year Prediction for Enterprise-Grade ML	2020	CIDR	7.0972826e-05
4,110	Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines	2020	SIGMOD	6.797656e-05
4,453	Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka	2021	SIGMOD	6.59179e-05
4,615	Efficient Placement of Decomposable Aggregation Functions for Stream Processing over Large Geo-Distributed Topologies	2024	VLDB	6.4985947e-05
5,939	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB	5.9378976e-05
6,270	Beyond Analytics: The Evolution of Stream Processing Systems	2020	SIGMOD	5.8275729e-05
6,903	Seagull: An Infrastructure for Load Prediction and Optimized Resource Allocation	2021	VLDB	5.6493582e-05
7,413	POLAR: Adaptive and Non-invasive Join Order Selection via Plans of Least Resistance	2024	VLDB	5.5316834e-05
7,813	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.4474823e-05
8,357	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3463255e-05
8,410	Spur: Mitigating Slow Instances in Large-Scale Streaming Pipelines	2020	SIGMOD	5.3360542e-05
9,482	Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing	2019	SIGMOD	5.1687151e-05
9,820	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB	5.1233734e-05
9,965	Lorentz: Learned SKU Recommendation Using Profile Data (DMDS)	2024	SIGMOD	5.1014161e-05
10,089	ContTune: Continuous Tuning by Conservative Bayesian Optimization for Distributed Stream Data Processing Systems	2023	VLDB	5.0806786e-05
10,552	Accelerating Stream Processing Engines via Hardware Offloading	2026	SIGMOD	4.9769913e-05
10,739	Scarf: Self-Adaptive Tuning via Multi-Objective Reinforcement Learning for Apache Flink	2026	VLDB	4.9769913e-05
10,743	APEROL: Adaptive Parallel Edge-to-cloud Runtime Optimization for Layered Workflow Execution	2026	VLDB	4.9769913e-05
10,855	Meerkat: Scalable, Network-Aware Failure Recovery for the Internet of Things	2026	VLDB	4.9769913e-05
11,208	Styx: Transactional Stateful Functions on Streaming Dataflows	2025	SIGMOD	4.9769913e-05
11,219	Scalable Complex Event Processing on Video Streams	2025	SIGMOD	4.9769913e-05
11,565	Intelligent Pooling: Proactive Resource Provisioning in Large-scale Cloud Service	2024	VLDB	4.9769913e-05
12,196	Minimizing Cost by Reducing Scaling Operations in Distributed Stream Processing	2019	VLDB	4.9769913e-05
12,238	Challenges and Experiences in Building an Efficient Apache Beam Runner For IBM Streams	2018	VLDB	4.9769913e-05

Outgoing Citations (Sorted by Pagerank)

Showing 7 of 7 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
127	The Design of the Borealis Stream Processing Engine	2005	CIDR	0.00030414379
218	MillWheel: Fault-Tolerant Stream Processing at Internet Scale	2013	VLDB	0.00024379041
224	Self-Driving Database Management Systems	2017	CIDR	0.00024011047
231	Storm @Twitter	2014	SIGMOD	0.00023830094
606	Twitter Heron: Stream Processing at Scale	2015	SIGMOD	0.00015627966
892	Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance	2010	VLDB	0.00013221727
940	Starfish: A Self-tuning System for Big Data Analytics	2011	CIDR	0.00012959992

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	231	Storm @Twitter	2014	SIGMOD
2	9,641	Shrink - Prescribing Resiliency Solutions for Streaming	2017	VLDB
3	7,317	Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning	2018	VLDB
4	3,034	Chi: A Scalable and Programmable Control Plane for Distributed Stream Processing Systems	2018	VLDB
5	11,138	Oceanus: Enable SLO-Aware Vertical Autoscaling for Cloud-Native Streaming Services in Tencent	2025	SIGMOD
6	10,162	Railgun: managing large streaming windows under MAD requirements	2021	VLDB
7	5,939	StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance	2023	VLDB
8	9,209	Dalton: Learned Partitioning for Distributed Data Streams	2023	VLDB
9	606	Twitter Heron: Stream Processing at Scale	2015	SIGMOD
10	9,820	Dhalion in Action: Automatic Management of Streaming Applications	2018	VLDB