SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training
Summary: SDP_PIPE targets heterogeneous cloud clusters for pipeline-parallel training, avoiding parameter-server bottlenecks and All-Reduce stragglers. It semi-decentralizes synchronization while centralizing global group scheduling, improving scalability, heterogeneity tolerance, and convergence. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Xupeng Miao (Carnegie Mellon University)
- 2. Yining Shi (Peking University)
- 3. Zhi Yang (Peking University)
- 4. Bin Cui (Peking University)
- 5. Zhihao Jia (Carnegie Mellon University)
BibTeX Citation
@article{miao_vldb23,
title = {{SDP\_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training}},
author = {Miao, Xupeng and Shi, Yining and Yang, Zhi and Cui, Bin and Jia, Zhihao},
journal = {PVLDB},
series = {{VLDB} '23},
volume = {16},
number = {9},
pages = {2354--2363},
doi = {10.14778/3598581.3598604},
url = {https://doi.org/10.14778/3598581.3598604},
year = {2023}
}Incoming Citations (Sorted by Pagerank)
Showing 3 of 3 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6,900 | Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving | 2025 | SIGMOD | 5.7430032e-05 |
| 9,880 | MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training | 2025 | SIGMOD | 5.2040783e-05 |
| 10,596 | NeutronCloud: Resource-Aware Distributed GNN Training in Fluctuating Cloud Environments | 2026 | VLDB | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 9 of 9 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 521 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB | 0.0001713368 |
| 1,150 | DimmWitted: A Study of Main-Memory Statistical Analytics | 2014 | VLDB | 0.00011943462 |
| 2,162 | Heterogeneity-aware Distributed Parameter Servers | 2017 | SIGMOD | 9.0581831e-05 |
| 2,485 | HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework | 2022 | VLDB | 8.5145736e-05 |
| 3,169 | Towards Demystifying Serverless Machine Learning Training | 2021 | SIGMOD | 7.6715222e-05 |
| 4,956 | Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce | 2021 | SIGMOD | 6.4290135e-05 |
| 5,003 | Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism | 2023 | VLDB | 6.4065691e-05 |
| 8,034 | Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent | 2023 | VLDB | 5.502946e-05 |
| 8,883 | FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement | 2023 | SIGMOD | 5.351513e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 3,764 | FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline | 2023 | VLDB |
| 2 | 4,799 | Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models | 2017 | VLDB |
| 3 | 8,905 | TensorSocket: Shared Data Loading for Deep Learning Training | 2026 | SIGMOD |
| 4 | 9,414 | Model-Parallel Model Selection for Deep Learning Systems | 2021 | SIGMOD |
| 5 | 10,887 | PipeTGL: (Near) Zero Bubble Memory-based Temporal Graph Neural Network Training via Pipeline Optimization | 2025 | VLDB |
| 6 | 10,219 | DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization | 2026 | SIGMOD |
| 7 | 521 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB |
| 8 | 2,473 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | VLDB |
| 9 | 2,162 | Heterogeneity-aware Distributed Parameter Servers | 2017 | SIGMOD |
| 10 | 4,956 | Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce | 2021 | SIGMOD |