Phoebe: A Learning-based Checkpoint Optimizer

Summary: Phoebe, a learning-based checkpoint optimizer, uses predictors (exec time, output size, start/end) to decompose plans and place checkpoints. Formulated as an integer program with a scalable heuristic, it minimizes hotspot storage and speeds restarts. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12613
Venue: VLDB
Year: 2021
Pagerank: 5.3035811e-05
Overall Rank: 9,224 | 36.72%
DOI: 10.14778/3476249.3476298

Incoming Non-self Citations Over Time

Authors

1. Yiwen Zhu (Microsoft)
2. Matteo Interlandi (Microsoft)
3. Abhishek Roy (Microsoft)
4. Krishnadhan Das (Microsoft)
5. Hiren Patel (Microsoft)
6. Malay Bag (Meta)
7. Hitesh Sharma (Google)
8. Alekh Jindal (Microsoft)

BibTeX Citation

@article{zhu_vldb21,
        title = {{Phoebe: A Learning-based Checkpoint Optimizer}},
        author = {Zhu, Yiwen and Interlandi, Matteo and Roy, Abhishek and Das, Krishnadhan and Patel, Hiren and Bag, Malay and Sharma, Hitesh and Jindal, Alekh},
        journal = {PVLDB},
        series = {{VLDB} '21},
        volume = {14},
        number = {11},
        pages = {2505--2518},
        doi = {10.14778/3476249.3476298},
        url = {https://doi.org/10.14778/3476249.3476298},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 8 of 8 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,998	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD	6.9676473e-05
5,388	Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing	2022	VLDB	6.2362811e-05
7,661	Machine Learning for Cloud Data Systems: the Progress so far and the Path Forward	2021	VLDB	5.5736026e-05
7,762	Runtime Variation in Big Data Analytics	2023	SIGMOD	5.5501898e-05
8,193	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.4696038e-05
8,861	Optimizing the cloud? Don't train models. Build oracles!	2024	CIDR	5.355716e-05
8,864	Pipemizer: An Optimizer for Analytics Data Pipelines	2022	VLDB	5.355022e-05
9,771	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD	5.2209769e-05

Outgoing Citations (Sorted by Pagerank)

Showing 19 of 19 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
30	SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets	2008	VLDB	0.00051174276
32	Hive - A Warehousing Solution Over a Map-Reduce Framework	2009	VLDB	0.00050111008
84	Learned Cardinalities: Estimating Correlated Joins with Deep Learning	2019	CIDR	0.00035838391
154	Neo: A Learned Query Optimizer	2019	VLDB	0.00028726181
323	DeepDB: Learn from Data, not from Queries!	2020	VLDB	0.00021264788
388	Preventing Bad Plans by Bounding the Impact of Cardinality Estimation Errors	2009	VLDB	0.00019410042
563	Plan-Structured Deep Neural Network Models for Query Performance Prediction	2019	VLDB	0.0001650812
697	Selectivity Estimation for Range Predicates using Lightweight Models	2019	VLDB	0.00014888851
983	Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management	2013	SIGMOD	0.0001283214
1,468	Towards a Learning Optimizer for Shared Clouds	2019	VLDB	0.00010686496
1,765	Selecting Subexpressions to Materialize at Datacenter Scale	2018	VLDB	9.8079546e-05
1,789	Fault-Tolerance in the Borealis Distributed Stream Processing System	2005	SIGMOD	9.7502324e-05
2,477	Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics	2017	SIGMOD	8.5239378e-05
2,622	A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans	2011	SIGMOD	8.3330136e-05
2,822	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD	8.0898536e-05
3,605	Computation Reuse in Analytics Job Service at Microsoft	2018	SIGMOD	7.2640711e-05
3,776	Fault-tolerant Stream Processing using a Distributed, Replicated File System	2008	VLDB	7.1347542e-05
6,194	Incorporating Super-Operators in Big-Data Query Optimizers	2020	VLDB	5.9470844e-05
9,596	Cost-based Fault-tolerance for Parallel Data Processing	2015	SIGMOD	5.2496236e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,394	Leveraging Query Logs and Machine Learning for Parametric Query Optimization	2022	VLDB
2	9,971	Athena: An Effective Learning-based Framework for Query Optimizer Performance Improvement	2025	SIGMOD
3	295	Accelerating Machine Learning Inference with Probabilistic Predicates	2018	SIGMOD
4	10,485	PTO: A Workload-driven Predictive Table Optimizer for Lakehouse Systems	2026	SIGMOD
5	8,615	A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning	2024	VLDB
6	1,904	CoPhy: A Scalable, Portable, and Interactive Index Advisor for Large Workloads	2011	VLDB
7	9,771	Rockhopper: A Robust Optimizer for Spark Configuration Tuning in Production Environment	2025	SIGMOD
8	11,290	Presto’s History-based Query Optimizer	2024	VLDB
9	2,822	Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings	2020	SIGMOD
10	5,059	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD