Database Paper Browser

Back to papers

Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism

Summary: Galvatron automatically explores a large hybrid-parallelism space (data/model/pipeline/tensor) to optimize multi‑GPU Transformer training via decision-tree decomposition and a dynamic-programming search. Outperforms prior limited-parallelism systems across varying GPU memory budgets. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID
13304
Venue
VLDB
Year
2023
Pagerank
5.0911095e-05
Overall Rank
6,377 | 55.64%
DOI
10.14778/3570690.3570697

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 13 of 13 citing papers.

Rank Citing Paper Year Venue Pagerank
3,995 How Large Language Models Will Disrupt Data Management 2023 VLDB 6.5513237e-05
7,152 Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity 2024 VLDB 4.8154191e-05
7,536 Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent 2023 VLDB 4.7176331e-05
8,126 SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training 2023 VLDB 4.5796615e-05
8,808 FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement 2023 SIGMOD 4.4454035e-05
9,326 BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach 2023 SIGMOD 4.3556432e-05
9,677 Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving 2025 SIGMOD 4.3047774e-05
9,694 EinDecomp: Decomposition of Declaratively-Specified Machine Learning and Numerical Computations for Parallel Execution 2025 VLDB 4.3025567e-05
9,805 MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training 2025 SIGMOD 4.2805224e-05
10,089 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 4.1945683e-05
10,492 Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization 2025 SIGMOD 4.1945683e-05
10,626 LobRA: Multi-tenant Fine-tuning over Heterogeneous Data 2025 VLDB 4.1945683e-05
11,059 DARKER: Efficient Transformer with Data-driven Attention Mechanism for Time Series 2024 VLDB 4.1945683e-05
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 5 of 5 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
411 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB 0.00023906921
683 Cerebro: A Data System for Optimized Deep Learning Model Selection 2020 VLDB 0.00018195476
2,677 HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework 2022 VLDB 8.3268401e-05
5,052 HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training 2022 SIGMOD 5.7337977e-05
5,333 Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce 2021 SIGMOD 5.5656575e-05
Previous Page 1 / 1 Next

Semantically Similar Papers

Overall Rank Paper Year Venue Pagerank
9,264 Model-Parallel Model Selection for Deep Learning Systems 2021 SIGMOD 4.3675421e-05
9,395 NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism 2025 VLDB 4.3441378e-05
9,596 Scalable Graph Convolutional Network Training on Distributed-Memory Systems 2023 VLDB 4.319218e-05
3,087 Scalable and Efficient Full-Graph GNN Training for Large Graphs 2023 SIGMOD 7.5939896e-05
411 PyTorch Distributed: Experiences on Accelerating Data Parallel Training 2020 VLDB 0.00023906921
3,363 CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers 2019 VLDB 7.1731921e-05
8,348 FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation 2024 VLDB 4.5410024e-05
10,089 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 4.1945683e-05
4,180 FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline 2023 VLDB 6.3793352e-05
9,603 Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads 2024 VLDB 4.3177432e-05