Phoebe: A Learning-based Checkpoint Optimizer

Summary: Phoebe, a learning-based checkpoint optimizer, uses predictors (exec time, output size, start/end) to decompose plans and place checkpoints. Formulated as an integer program with a scalable heuristic, it minimizes hotspot storage and speeds restarts. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h88ed67fdeba5227f
Venue: VLDB
Year: 2021
Pagerank: 5.1845217e-05
Overall Rank: 9,367 | 37.05%
DOI: 10.14778/3476249.3476298
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Matteo Interlandi (Microsoft)
3. Abhishek Roy (Microsoft)
4. Krishnadhan Das (Microsoft)
5. Hiren Patel (Microsoft)
6. Malay Bag (Meta)
7. Hitesh Sharma (Google)
8. Alekh Jindal (Microsoft)

BibTeX Citation

@article{zhu_vldb21,
        title = {{Phoebe: A Learning-based Checkpoint Optimizer}},
        author = {Zhu, Yiwen and Interlandi, Matteo and Roy, Abhishek and Das, Krishnadhan and Patel, Hiren and Bag, Malay and Sharma, Hitesh and Jindal, Alekh},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {11},
        pages = {2505--2518},
        doi = {10.14778/3476249.3476298},
        url = {https://doi.org/10.14778/3476249.3476298},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,948	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	6.9051584e-05
5,062	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	6.2896995e-05
7,813	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.4474823e-05
7,930	Runtime Variation in Big Data Analytics	2023	SIGMOD	5.42309e-05
8,357	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3463255e-05
8,978	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.2446911e-05
9,028	Optimizing the cloud? Don't train models. Build oracles!	2024	CIDR	5.2330697e-05
9,031	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.2323895e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050475202
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049821554
85	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035876108
144	Neo: A Learned Query Optimizer	2019	VLDB	0.00029090793
318	DeepDB: Learn from Data, not from Queries!	2020	VLDB	0.00021166957
386	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019446558
560	Plan-Structured Deep Neural Network Models for Query Performance Prediction	2019	VLDB	0.00016408613
691	Selectivity Estimation for Range Predicates using Lightweight Models	2019	VLDB	0.00014737455
970	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD	0.00012776109
1,432	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	0.00010676754
1,747	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.7303647e-05
1,829	Fault-Tolerance in the Borealis Distributed Stream Processing System	2005	SIGMOD	9.5470848e-05
2,448	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4508839e-05
2,673	A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans	2011	SIGMOD	8.1453443e-05
2,833	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9539771e-05
3,544	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2108612e-05
3,792	Fault-tolerant Stream Processing using a Distributed, Replicated File System	2008	VLDB	7.0157179e-05
6,293	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	5.8195888e-05
9,778	Cost-based Fault-tolerance for Parallel Data Processing	2015	SIGMOD	5.1294772e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	1,432	Towards a Learning Optimizer for Shared Clouds	2019	VLDB
2	9,556	Athena: An Effective Learning-based Framework for Query Optimizer Performance Improvement	2025	SIGMOD
3	282	Accelerating Machine Learning Inference with Probabilistic Predicates	2018	SIGMOD
4	10,683	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD
5	8,009	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
6	1,815	CoPhy: A Scalable, Portable, and Interactive Index Advisor for Large Workloads	2011	VLDB
7	9,031	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD
8	9,122	Presto’s History-based Query Optimizer	2024	VLDB
9	2,833	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
10	5,110	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD