DBScholar

Back to papers

PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Summary: Design, implementation, and evaluation of PyTorch Distributed Data Parallel for scalable data-parallel training on GPUs. Unique practical optimizations—gradient bucketing, compute/communication overlap, and skipped gradient sync—achieving near-linear scaling to 256 GPUs. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID
12374
Venue
VLDB
Year
2020
Pagerank
0.0001713368
Overall Rank
521 | 96.43%
DOI
10.14778/3415478.3415530

Incoming Non-self Citations Over Time

Authors

BibTeX Citation

@article{li_vldb20,
        title = {{PyTorch Distributed: Experiences on Accelerating Data Parallel Training}},
        author = {Li, Shen and Zhao, Yanli and Varma, Rohan and Salpekar, Omkar and Noordhuis, Pieter and Li, Teng and Paszke, Adam and Smith, Jeff and Vaughan, Brian and Damania, Pritam and Chintala, Soumith},
        journal = {PVLDB},
        series = {{VLDB} '20},
        volume = {13},
        number = {12},
        pages = {3005--3018},
        doi = {10.14778/3415478.3415530},
        url = {https://doi.org/10.14778/3415478.3415530},
        year = {2020}
}

Incoming Citations (Sorted by Pagerank)

Showing 29 of 29 citing papers.

Rank Citing Paper Year Venue Pagerank
2,473 PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel 2023 VLDB 8.5326287e-05
2,485 HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework 2022 VLDB 8.5145736e-05
2,695 NeutronStar: Distributed GNN Training with Hybrid Dependency Management 2022 SIGMOD 8.2468134e-05
2,823 Query Processing on Tensor Computation Runtimes 2022 VLDB 8.0893814e-05
3,169 Towards Demystifying Serverless Machine Learning Training 2021 SIGMOD 7.6715222e-05
4,912 HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training 2022 SIGMOD 6.4481656e-05
4,956 Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce 2021 SIGMOD 6.4290135e-05
5,003 Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism 2023 VLDB 6.4065691e-05
5,195 Tensor Relational Algebra for Distributed Machine Learning System Design 2021 VLDB 6.3232105e-05
5,876 BAGUA: Scaling up Distributed Learning with System Relaxations 2022 VLDB 6.0557672e-05
7,075 Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity 2024 VLDB 5.7099047e-05
8,101 SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training 2023 VLDB 5.4870581e-05
8,637 Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers 2022 VLDB 5.3954887e-05
8,725 mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs 2025 VLDB 5.3766157e-05
8,855 ANN Softmax: Acceleration of Extreme Classification Training 2022 VLDB 5.3573227e-05
8,883 FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement 2023 SIGMOD 5.351513e-05
8,979 Cerebro: A Layered Data Platform for Scalable Deep Learning 2021 CIDR 5.3399615e-05
9,371 Towards an Optimized GROUP BY Abstraction for Large-Scale Machine Learning 2021 VLDB 5.275595e-05
9,469 How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study 2024 VLDB 5.2634238e-05
9,475 BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach 2023 SIGMOD 5.2634238e-05
9,737 Saturn: An Optimized Data System for Multi-Large-Model Deep Learning Workloads 2024 VLDB 5.227679e-05
9,834 EinDecomp: Decomposition of Declaratively-Specified Machine Learning and Numerical Computations for Parallel Execution 2025 VLDB 5.2112879e-05
10,219 DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization 2026 SIGMOD 5.093636e-05
10,380 Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment 2026 SIGMOD 5.093636e-05
10,844 GPEmu: A GPU Emulator for Faster and Cheaper Prototyping and Evaluation of Deep Learning System Research 2025 VLDB 5.093636e-05
10,880 LobRA: Multi-tenant Fine-tuning over Heterogeneous Data 2025 VLDB 5.093636e-05
10,890 Heta: Distributed Training of Heterogeneous Graph Neural Networks 2025 VLDB 5.093636e-05
10,907 Effective and Efficient Distributed Temporal Graph Learning through Hotspot Memory Sharing 2025 VLDB 5.093636e-05
13,327 DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems 2025 VLDB -
Previous Page 1 / 1 Next

Outgoing Citations (Sorted by Pagerank)

Showing 0 of 0 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank Cited Paper Year Venue Pagerank
Previous Page 1 / 1 Next

Semantically Similar Papers

# Overall Rank Paper Year Venue
1 6,519 Dynamic Parameter Allocation in Parameter Servers 2020 VLDB
2 1,234 Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture 2021 VLDB
3 5,305 Parallel Training of Knowledge Graph Embedding Models: A Comparison of Techniques 2022 VLDB
4 9,546 NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism 2025 VLDB
5 8,101 SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training 2023 VLDB
6 9,729 Scalable Graph Convolutional Network Training on Distributed-Memory Systems 2023 VLDB
7 8,905 TensorSocket: Shared Data Loading for Deep Learning Training 2026 SIGMOD
8 10,113 Distributed Learning of Fully Connected Neural Networks using Independent Subnet Training 2022 VLDB
9 4,956 Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce 2021 SIGMOD
10 2,473 PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel 2023 VLDB