Phoebe: A Learning-based Checkpoint Optimizer

Summary: Phoebe, a learning-based checkpoint optimizer, uses predictors (exec time, output size, start/end) to decompose plans and place checkpoints. Formulated as an integer program with a scalable heuristic, it minimizes hotspot storage and speeds restarts. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h88ed67fdeba5227f
Venue: VLDB
Year: 2021
Pagerank: 5.1676892e-05
Overall Rank: 9,394 | 37.04%
DOI: 10.14778/3476249.3476298
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Matteo Interlandi (Microsoft)
3. Abhishek Roy (Microsoft)
4. Krishnadhan Das (Microsoft)
5. Hiren Patel (Microsoft)
6. Malay Bag (Meta)
7. Hitesh Sharma (Google)
8. Alekh Jindal (Microsoft)

BibTeX Citation

@article{zhu_vldb21,
        title = {{Phoebe: A Learning-based Checkpoint Optimizer}},
        author = {Zhu, Yiwen and Interlandi, Matteo and Roy, Abhishek and Das, Krishnadhan and Patel, Hiren and Bag, Malay and Sharma, Hitesh and Jindal, Alekh},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {11},
        pages = {2505--2518},
        doi = {10.14778/3476249.3476298},
        url = {https://doi.org/10.14778/3476249.3476298},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,951	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	6.8884038e-05
5,043	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	6.28918e-05
7,830	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.4294933e-05
7,944	Runtime Variation in Big Data Analytics	2023	SIGMOD	5.4051579e-05
8,372	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3287108e-05
8,998	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.2273528e-05
9,041	Optimizing the cloud? Don't train models. Build oracles!	2024	CIDR	5.2183053e-05
9,052	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.215088e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050364918
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049707539
84	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035920657
142	Neo: A Learned Query Optimizer	2019	VLDB	0.00029114937
320	DeepDB: Learn from Data, not from Queries!	2020	VLDB	0.00021128745
381	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019497061
545	Plan-Structured Deep Neural Network Models for Query Performance Prediction	2019	VLDB	0.0001657599
690	Selectivity Estimation for Range Predicates using Lightweight Models	2019	VLDB	0.00014703258
973	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD	0.00012734154
1,436	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	0.00010658836
1,757	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.7050769e-05
1,841	Fault-Tolerance in the Borealis Distributed Stream Processing System	2005	SIGMOD	9.5155524e-05
2,452	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4280184e-05
2,684	A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans	2011	SIGMOD	8.1185673e-05
2,828	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9476509e-05
3,550	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.1895215e-05
3,801	Fault-tolerant Stream Processing using a Distributed, Replicated File System	2008	VLDB	6.9925771e-05
6,294	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	5.8043081e-05
9,809	Cost-based Fault-tolerance for Parallel Data Processing	2015	SIGMOD	5.1125251e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	9,583	Athena: An Effective Learning-based Framework for Query Optimizer Performance Improvement	2025	SIGMOD
2	1,436	Towards a Learning Optimizer for Shared Clouds	2019	VLDB
3	278	Accelerating Machine Learning Inference with Probabilistic Predicates	2018	SIGMOD
4	10,744	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD
5	8,026	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
6	1,823	CoPhy: A Scalable, Portable, and Interactive Index Advisor for Large Workloads	2011	VLDB
7	9,052	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD
8	9,141	Presto’s History-based Query Optimizer	2024	VLDB
9	2,828	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
10	5,126	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD