SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training
Summary: SDP_PIPE targets heterogeneous cloud clusters for pipeline-parallel training, avoiding parameter-server bottlenecks and All-Reduce stragglers. It semi-decentralizes synchronization while centralizing global group scheduling, improving scalability, heterogeneity tolerance, and convergence. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Xupeng Miao (Carnegie Mellon University)
- 2. Yining Shi (Peking University)
- 3. Zhi Yang (Peking University)
- 4. Bin Cui (Peking University)
- 5. Zhihao Jia (Carnegie Mellon University)
BibTeX Citation
@article{miao_vldb23,
title = {{SDP\_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training}},
author = {Miao, Xupeng and Shi, Yining and Yang, Zhi and Cui, Bin and Jia, Zhihao},
journal = {PVLDB},
series = {{VLDB} '23},
volume = {16},
number = {9},
pages = {2354--2363},
doi = {10.14778/3598581.3598604},
url = {https://doi.org/10.14778/3598581.3598604},
year = {2023}
}Incoming Citations (Sorted by Pagerank)
Showing 3 of 3 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 6,166 | Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving | 2025 | SIGMOD | 5.8631131e-05 |
| 10,042 | MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training | 2025 | SIGMOD | 5.0921006e-05 |
| 11,043 | NeutronCloud: Resource-Aware Distributed GNN Training in Fluctuating Cloud Environments | 2026 | VLDB | 4.9793485e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 9 of 9 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 522 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB | 0.00016912378 |
| 1,167 | DimmWitted: A Study of Main-Memory Statistical Analytics | 2014 | VLDB | 0.00011729888 |
| 2,184 | Heterogeneity-aware Distributed Parameter Servers | 2017 | SIGMOD | 8.8958335e-05 |
| 2,519 | HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework | 2022 | VLDB | 8.3521095e-05 |
| 3,240 | Towards Demystifying Serverless Machine Learning Training | 2021 | SIGMOD | 7.5002772e-05 |
| 4,963 | Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce | 2021 | SIGMOD | 6.3384091e-05 |
| 5,009 | Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism | 2023 | VLDB | 6.3158866e-05 |
| 8,197 | Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent | 2023 | VLDB | 5.3794747e-05 |
| 8,812 | FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement | 2023 | SIGMOD | 5.2722828e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 3,811 | FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline | 2023 | VLDB |
| 2 | 4,901 | Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models | 2017 | VLDB |
| 3 | 9,067 | TensorSocket: Shared Data Loading for Deep Learning Training | 2026 | SIGMOD |
| 4 | 9,595 | Model-Parallel Model Selection for Deep Learning Systems | 2021 | SIGMOD |
| 5 | 9,232 | PipeTGL: (Near) Zero Bubble Memory-based Temporal Graph Neural Network Training via Pipeline Optimization | 2025 | VLDB |
| 6 | 10,435 | DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization | 2026 | SIGMOD |
| 7 | 522 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB |
| 8 | 2,246 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | VLDB |
| 9 | 2,184 | Heterogeneity-aware Distributed Parameter Servers | 2017 | SIGMOD |
| 10 | 4,963 | Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce | 2021 | SIGMOD |