Phoebe: A Learning-based Checkpoint Optimizer

Summary: Phoebe, a learning-based checkpoint optimizer, uses predictors (exec time, output size, start/end) to decompose plans and place checkpoints. Formulated as an integer program with a scalable heuristic, it minimizes hotspot storage and speeds restarts. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: h88ed67fdeba5227f
Venue: VLDB
Year: 2021
Pagerank: 5.1869771e-05
Overall Rank: 9,358 | 37.09%
DOI: 10.14778/3476249.3476298

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Matteo Interlandi (Microsoft)
3. Abhishek Roy (Microsoft)
4. Krishnadhan Das (Microsoft)
5. Hiren Patel (Microsoft)
6. Malay Bag (Meta)
7. Hitesh Sharma (Google)
8. Alekh Jindal (Microsoft)

BibTeX Citation

@article{zhu_vldb21,
        title = {{Phoebe: A Learning-based Checkpoint Optimizer}},
        author = {Zhu, Yiwen and Interlandi, Matteo and Roy, Abhishek and Das, Krishnadhan and Patel, Hiren and Bag, Malay and Sharma, Hitesh and Jindal, Alekh},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {11},
        pages = {2505--2518},
        doi = {10.14778/3476249.3476298},
        url = {https://doi.org/10.14778/3476249.3476298},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,949	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	6.9052796e-05
5,058	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	6.2926774e-05
7,806	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.4500623e-05
7,925	Runtime Variation in Big Data Analytics	2023	SIGMOD	5.4256585e-05
8,352	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.3488341e-05
8,968	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.2471751e-05
9,020	Optimizing the cloud? Don't train models. Build oracles!	2024	CIDR	5.2355482e-05
9,023	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.2348677e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00050495102
31	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00049839909
85	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035864347
145	Neo: A Learned Query Optimizer	2019	VLDB	0.0002908188
318	DeepDB: Learn from Data, not from Queries!	2020	VLDB	0.00021167555
386	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019444411
560	Plan-Structured Deep Neural Network Models for Query Performance Prediction	2019	VLDB	0.00016403151
692	Selectivity Estimation for Range Predicates using Lightweight Models	2019	VLDB	0.00014741011
968	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD	0.00012782051
1,433	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	0.00010677711
1,745	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.7343818e-05
1,828	Fault-Tolerance in the Borealis Distributed Stream Processing System	2005	SIGMOD	9.5516021e-05
2,446	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.4547121e-05
2,672	A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans	2011	SIGMOD	8.1491683e-05
2,834	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	7.9560627e-05
3,545	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2134803e-05
3,790	Fault-tolerant Stream Processing using a Distributed, Replicated File System	2008	VLDB	7.0190297e-05
6,290	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	5.8223076e-05
9,773	Cost-based Fault-tolerance for Parallel Data Processing	2015	SIGMOD	5.1319066e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,039	Leveraging Query Logs and Machine Learning for Parametric Query Optimization	2022	VLDB
2	1,433	Towards a Learning Optimizer for Shared Clouds	2019	VLDB
3	281	Accelerating Machine Learning Inference with Probabilistic Predicates	2018	SIGMOD
4	10,672	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD
5	8,004	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
6	1,815	CoPhy: A Scalable, Portable, and Interactive Index Advisor for Large Workloads	2011	VLDB
7	9,023	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD
8	9,113	Presto’s History-based Query Optimizer	2024	VLDB
9	2,834	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
10	5,110	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD