Pipemizer: An Optimizer for Analytics Data Pipelines

Summary: Pipemizer: an optimizer and recommender for analytics data pipelines. Introduces pipeline-aware statistics, inter-job operator push-up, and split/merge optimizations to boost cross-job performance; demonstrated on large-scale SCOPE workloads with 650k daily jobs and 70% inter-job dependencies, enabling automated recommendations. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 13057
Venue: VLDB
Year: 2022
Pagerank: 5.355022e-05
Overall Rank: 8,864 | 39.19%
DOI: 10.14778/3554821.3554881

Incoming Non-self Citations Over Time

Authors

1. Sunny Gakhar (Microsoft)
2. Joyce Cahoon (Microsoft)
3. Wangchao Le (Microsoft)
4. Xiangnan Li (Microsoft)
5. Kaushik Ravichandran (Microsoft)
6. Hiren Patel (Microsoft)
7. Marc Friedman (Microsoft)
8. Brandon Haynes (Microsoft)
9. Shi Qiao (Keebo; Microsoft)
10. Alekh Jindal (Keebo; Microsoft)
11. Jyoti Leeka (Microsoft)

BibTeX Citation

@article{gakhar_vldb22,
        title = {{Pipemizer: An Optimizer for Analytics Data Pipelines}},
        author = {Gakhar, Sunny and Cahoon, Joyce and Le, Wangchao and Li, Xiangnan and Ravichandran, Kaushik and Patel, Hiren and Friedman, Marc and Haynes, Brandon and Qiao, Shi and Jindal, Alekh and Leeka, Jyoti},
        journal = {PVLDB},
        series = {{VLDB} '22},
        volume = {15},
        number = {12},
        pages = {3710--3713},
        doi = {10.14778/3554821.3554881},
        url = {https://doi.org/10.14778/3554821.3554881},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 3 of 3 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
8,193	Towards Building Autonomous Data Services on Azure	2023	SIGMOD	5.4696038e-05
8,448	Towards Query Optimizer as a Service (QOaaS) in a Unified LakeHouse Ecosystem: Can One QO Rule Them All?	2025	CIDR	5.4235725e-05
11,150	Proactive Resume and Pause of Resources for Microsoft Azure SQL Database Serverless	2024	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 9 of 9 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
128	Efficient and Extensible Algorithms for Multi Query Optimization	2000	SIGMOD	0.0003072825
387	AutoAdmin "What-if" Index Analysis Utility	1998	SIGMOD	0.00019442332
822	Pipelining in Multi-Query Optimization	2001	PODS	0.00013807229
1,925	POLARIS: The Distributed SQL Engine in Azure Synapse	2020	VLDB	9.4764398e-05
2,657	Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities	2021	SIGMOD	8.2887895e-05
2,722	Automatic Physical Design Tuning: Workload as a Sequence	2006	SIGMOD	8.206711e-05
3,163	Multi-Query Optimization in MapReduce Framework	2014	VLDB	7.6784171e-05
6,121	The Cosmos Big Data Platform at Microsoft: Over a Decade of Progress and a Decade to Look Forward	2021	VLDB	5.9688569e-05
9,224	Phoebe: A Learning-based Checkpoint Optimizer	2021	VLDB	5.3035811e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	11,207	Pasta: A Cost-Based Optimizer for Generating Pipelining Schedules for Dataflow DAGs	2024	SIGMOD
2	8,177	HAIPipe: Combining Human-generated and Machine-generated Pipelines for Data Preparation	2023	SIGMOD
3	383	QPipe: A Simultaneously Pipelined Relational Query Engine	2005	SIGMOD
4	6,309	Materialization and Reuse Optimizations for Production Data Science Pipelines	2022	SIGMOD
5	8,465	Predicate Pushdown for Data Science Pipelines	2023	SIGMOD
6	4,875	Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search	2021	VLDB
7	5,059	Steering Query Optimizers: A Practical Take on Big Data Workloads	2021	SIGMOD
8	5,704	Optimizing Data Pipelines for Machine Learning in Feature Stores	2023	VLDB
9	6,255	Elastic Pipelining in an In-Memory Database Cluster	2016	SIGMOD
10	3,998	Deploying a Steered Query Optimizer in Production at Microsoft	2022	SIGMOD