PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Summary: Introduces PyTorch Fully Sharded Data Parallel (FSDP), an industry-grade, non-intrusive sharding framework co-designed with PyTorch internals (Tensor, dispatcher, CUDA allocator) to enable training of much larger models than DDP. FSDP bundles memory and communication optimizations across hardware to achieve near-linear TFLOPS scalability and DDP-comparable throughput while drastically reducing memory footprint. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: h0ecbdae73a7b577f
Venue: VLDB
Year: 2023
Pagerank: 8.7637336e-05
Overall Rank: 2,246 | 84.91%
DOI: 10.14778/3611540.3611569

Incoming Non-self Citations Over Time

Authors

1. Yanli Zhao (Meta)
2. Andrew Gu (Meta)
3. Rohan Varma (Meta)
4. Liang Luo (Meta)
5. Chien-Chin Huang (Meta)
6. Min Xu (Meta)
7. Less Wright (Meta)
8. Hamid Shojanazeri (Meta)
9. Myle Ott (Meta)
10. Sam Shleifer (Meta)
11. Alban Desmaison (Meta)
12. Can Balioglu (Meta)
13. Pritam Damania (Meta)
14. Bernard Nguyen (Meta)
15. Geeta Chauhan (Meta)
16. Yuchen Hao (Meta)
17. Ajit Mathews (Meta)
18. Shen Li (Meta)

BibTeX Citation

@article{zhao_vldb23,
        title = {{PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel}},
        author = {Zhao, Yanli and Gu, Andrew and Varma, Rohan and Luo, Liang and Huang, Chien-Chin and Xu, Min and Wright, Less and Shojanazeri, Hamid and Ott, Myle and Shleifer, Sam and Desmaison, Alban and Balioglu, Can and Damania, Pritam and Nguyen, Bernard and Chauhan, Geeta and Hao, Yuchen and Mathews, Ajit and Li, Shen},
        journal = {PVLDB},
        series = {{VLDB} '23},
        volume = {16},
        number = {12},
        pages = {3848--3860},
        doi = {10.14778/3611540.3611569},
        url = {https://doi.org/10.14778/3611540.3611569},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 13 of 13 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,250	How Large Language Models Will Disrupt Data Management	2023	VLDB	7.4938546e-05
8,887	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB	5.2559789e-05
10,042	MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training	2025	SIGMOD	5.0921006e-05
10,435	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD	4.9793485e-05
10,577	Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment	2026	SIGMOD	4.9793485e-05
10,659	Mixtera: A Data Plane for Foundation Model Training	2026	SIGMOD	4.9793485e-05
10,861	UniTG: A Unified System for Efficient and Seamless Textual Graph Learning	2026	VLDB	4.9793485e-05
11,124	Sequoia: An Accessible and Extensible Framework for Privacy-Preserving Machine Learning over Distributed Data	2025	SIGMOD	4.9793485e-05
11,189	Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization	2025	SIGMOD	4.9793485e-05
11,250	GPEmu: A GPU Emulator for Faster and Cheaper Prototyping and Evaluation of Deep Learning System Research	2025	VLDB	4.9793485e-05
11,282	LobRA: Multi-tenant Fine-tuning over Heterogeneous Data	2025	VLDB	4.9793485e-05
11,328	Evoschema: Towards Text-To-Sql Robustness Against Schema Evolution	2025	VLDB	4.9793485e-05
13,648	DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems	2025	VLDB	-

Outgoing Citations (Sorted by Pagerank)

Showing 2 of 2 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
522	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00016912378
3,450	MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud	2023	VLDB	7.2921762e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	10,341	Distributed Learning of Fully Connected Neural Networks using Independent Subnet Training	2022	VLDB
2	2,596	Scalable and Efficient Full-Graph GNN Training for Large Graphs	2023	SIGMOD
3	10,435	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD
4	3,811	FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline	2023	VLDB
5	10,801	PystachIO: Efficient Distributed GPU Query Processing with PyTorch over Fast Networks & Fast Storage	2026	VLDB
6	4,963	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD
7	9,595	Model-Parallel Model Selection for Deep Learning Systems	2021	SIGMOD
8	9,067	TensorSocket: Shared Data Loading for Deep Learning Training	2026	SIGMOD
9	8,251	SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB
10	522	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB