DBScholar

Back to papers

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Summary: Galvatron automatically searches hybrid Transformer parallelism across multiple GPUs and memory budgets. A decision-tree decomposition/pruning stage combined with dynamic programming efficiently navigates the large strategy space, outperforming prior limited-parallelism systems in throughput. (summarized by gpt-5.6-luna on Jul 24 2026)

Paper ID
hea38cacf948371f7
Venue
VLDB
Year
2023
Pagerank
6.3030066e-05
Overall Rank
5,011 | 66.42%
DOI
10.14778/3570690.3570697
PDF
Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{miao_vldb23,
        title = {{Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism}},
        author = {Miao, Xupeng and Wang, Yujie and Jiang, Youhe and Shi, Chunan and Nie, Xiaonan and Zhang, Hailin and Cui, Bin},
        journal = {PVLDB},
        series = {{VLDB} '23},
        volume = {16},
        number = {3},
        pages = {470--479},
        doi = {10.14778/3570690.3570697},
        url = {https://doi.org/10.14778/3570690.3570697},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 15 of 15 citing papers.

Rank Citing Paper Year Venue Pagerank
3,223 How Large Language Models Will Disrupt Data Management 2023 VLDB 7.5043163e-05
4,360 Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity 2024 VLDB 6.6274323e-05
4,910 Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving 2025 SIGMOD 6.3468554e-05
8,220 Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent 2023 VLDB 5.3591486e-05
8,228 SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training 2023 VLDB 5.3581744e-05
8,833 FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement 2023 SIGMOD 5.2528075e-05
9,691 BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach 2023 SIGMOD 5.1258854e-05
10,016 MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training 2025 SIGMOD 5.0825892e-05
10,062 EinDecomp: Decomposition of Declaratively-Specified Machine Learning and Numerical Computations for Parallel Execution 2025 VLDB 5.0751118e-05
10,410 StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training 2027 SIGMOD 4.9605343e-05
10,513 DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization 2026 SIGMOD 4.9605343e-05
10,650 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 4.9605343e-05
11,257 Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization 2025 SIGMOD 4.9605343e-05
11,346 LobRA: Multi-tenant Fine-tuning over Heterogeneous Data 2025 VLDB 4.9605343e-05
11,647 DARKER: Efficient Transformer with Data-driven Attention Mechanism for Time Series 2024 VLDB 4.9605343e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
526 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB 0.00016866695
1,154 Cerebro: A Data System for Optimized Deep Learning Model Selection 2020 VLDB 0.00011762688
2,525 HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework 2022 VLDB 8.3302991e-05
4,961 Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce 2021 SIGMOD 6.3246342e-05
4,974 HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training 2022 SIGMOD 6.3196773e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 8,174 NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism 2025 VLDB
2 9,942 Scalable Graph Convolutional Network Training on Distributed-Memory Systems 2023 VLDB
3 2,325 Scalable and Efficient Full-Graph GNN Training for Large Graphs 2023 SIGMOD
4 526 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB
5 3,770 CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers 2019 VLDB
6 10,513 DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization 2026 SIGMOD
7 10,650 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD
8 7,848 FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation 2024 VLDB
9 3,800 FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline 2023 VLDB
10 7,091 Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads 2024 VLDB