Database Paper Browser

Back to papers

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Summary: Galvatron automatically explores a large hybrid-parallelism space (data/model/pipeline/tensor) to optimize multi‑GPU Transformer training via decision-tree decomposition and a dynamic-programming search. Outperforms prior limited-parallelism systems across varying GPU memory budgets. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID
13305
Venue
VLDB
Year
2023
Pagerank
5.0903244e-05
Overall Rank
6,361 | 55.80%
DOI
10.14778/3570690.3570697

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 13 of 13 citing papers.

Rank Citing Paper Year Venue Pagerank
3,982 How Large Language Models Will Disrupt Data Management 2023 VLDB 6.5595332e-05
7,143 Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity 2024 VLDB 4.8143774e-05
7,535 Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent 2023 VLDB 4.7131061e-05
8,117 SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training 2023 VLDB 4.5788485e-05
8,807 FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement 2023 SIGMOD 4.4413307e-05
9,331 BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach 2023 SIGMOD 4.351469e-05
9,677 Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving 2025 SIGMOD 4.3006524e-05
9,693 EinDecomp: Decomposition of Declaratively-Specified Machine Learning and Numerical Computations for Parallel Execution 2025 VLDB 4.2984337e-05
9,786 MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training 2025 SIGMOD 4.2799988e-05
10,089 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 4.1905499e-05
10,502 Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization 2025 SIGMOD 4.1905499e-05
10,634 LobRA: Multi-tenant Fine-tuning over Heterogeneous Data 2025 VLDB 4.1905499e-05
11,062 DARKER: Efficient Transformer with Data-driven Attention Mechanism for Time Series 2024 VLDB 4.1905499e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
411 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB 0.00023881138
684 Cerebro: A Data System for Optimized Deep Learning Model Selection 2020 VLDB 0.00018152321
2,678 HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework 2022 VLDB 8.3224016e-05
5,169 HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training 2022 SIGMOD 5.642415e-05
5,332 Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce 2021 SIGMOD 5.5640779e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
9,269 Model-Parallel Model Selection for Deep Learning Systems 2021 SIGMOD 4.3633561e-05
9,400 NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism 2025 VLDB 4.3399748e-05
9,596 Scalable Graph Convolutional Network Training on Distributed-Memory Systems 2023 VLDB 4.3150788e-05
3,092 Scalable and Efficient Full-Graph GNN Training for Large Graphs 2023 SIGMOD 7.5869574e-05
411 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB 0.00023881138
3,372 CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers 2019 VLDB 7.1633912e-05
8,343 FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation 2024 VLDB 4.5366487e-05
10,089 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 4.1905499e-05
4,175 FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline 2023 VLDB 6.3772575e-05
9,603 Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads 2024 VLDB 4.3136057e-05