PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Summary: Introduces PyTorch Fully Sharded Data Parallel (FSDP), an industry-grade, non-intrusive sharding framework co-designed with PyTorch internals (Tensor, dispatcher, CUDA allocator) to enable training of much larger models than DDP. FSDP bundles memory and communication optimizations across hardware to achieve near-linear TFLOPS scalability and DDP-comparable throughput while drastically reducing memory footprint. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: h0ecbdae73a7b577f
Venue: VLDB
Year: 2023
Pagerank: 0.00010061487
Overall Rank: 1,612 | 89.20%
DOI: 10.14778/3611540.3611569
PDF: Download (CC BY-NC-ND 4.0)

Incoming Non-self Citations Over Time

Authors

1. Yanli Zhao (Meta)
2. Andrew Gu (Meta)
3. Rohan Varma (Meta)
4. Liang Luo (Meta)
5. Chien-Chin Huang (Meta)
6. Min Xu (Meta)
7. Less Wright (Meta)
8. Hamid Shojanazeri (Meta)
9. Myle Ott (Meta)
10. Sam Shleifer (Meta)
11. Alban Desmaison (Meta)
12. Can Balioglu (Meta)
13. Pritam Damania (Meta)
14. Bernard Nguyen (Meta)
15. Geeta Chauhan (Meta)
16. Yuchen Hao (Meta)
17. Ajit Mathews (Meta)
18. Shen Li (Meta)

BibTeX Citation

@article{zhao_vldb23,
        title = {{PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel}},
        author = {Zhao, Yanli and Gu, Andrew and Varma, Rohan and Luo, Liang and Huang, Chien-Chin and Xu, Min and Wright, Less and Shojanazeri, Hamid and Ott, Myle and Shleifer, Sam and Desmaison, Alban and Balioglu, Can and Damania, Pritam and Nguyen, Bernard and Chauhan, Geeta and Hao, Yuchen and Mathews, Ajit and Li, Shen},
        journal = {PVLDB},
        series = {{VLDB} '23},
        volume = {16},
        number = {12},
        pages = {3848--3860},
        doi = {10.14778/3611540.3611569},
        url = {https://doi.org/10.14778/3611540.3611569},
        year = {2023}
}

Incoming Citations (Sorted by Pagerank)

Showing 15 of 15 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,223	How Large Language Models Will Disrupt Data Management	2023	VLDB	7.5043163e-05
8,916	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB	5.2361195e-05
9,994	Evoschema: Towards Text-To-Sql Robustness Against Schema Evolution	2025	VLDB	5.0845476e-05
10,016	MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training	2025	SIGMOD	5.0825892e-05
10,410	StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training	2027	SIGMOD	4.9605343e-05
10,415	ZenFlow: Enabling Stall-Free Offloading for LLM Training	2027	SIGMOD	4.9605343e-05
10,513	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD	4.9605343e-05
10,650	Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment	2026	SIGMOD	4.9605343e-05
10,731	Mixtera: A Data Plane for Foundation Model Training	2026	SIGMOD	4.9605343e-05
10,931	UniTG: A Unified System for Efficient and Seamless Textual Graph Learning	2026	VLDB	4.9605343e-05
11,193	Sequoia: An Accessible and Extensible Framework for Privacy-Preserving Machine Learning over Distributed Data	2025	SIGMOD	4.9605343e-05
11,257	Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization	2025	SIGMOD	4.9605343e-05
11,314	GPEmu: A GPU Emulator for Faster and Cheaper Prototyping and Evaluation of Deep Learning System Research	2025	VLDB	4.9605343e-05
11,346	LobRA: Multi-tenant Fine-tuning over Heterogeneous Data	2025	VLDB	4.9605343e-05
13,694	DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems	2025	VLDB	-

Outgoing Citations (Sorted by Pagerank)

Showing 2 of 2 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
526	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00016866695
3,234	MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud	2023	VLDB	7.4879757e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	4,927	Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models	2017	VLDB
2	2,325	Scalable and Efficient Full-Graph GNN Training for Large Graphs	2023	SIGMOD
3	10,513	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD
4	3,800	FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline	2023	VLDB
5	10,872	PystachIO: Efficient Distributed GPU Query Processing with PyTorch over Fast Networks & Fast Storage	2026	VLDB
6	4,961	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD
7	9,629	Model-Parallel Model Selection for Deep Learning Systems	2021	SIGMOD
8	9,097	TensorSocket: Shared Data Loading for Deep Learning Training	2026	SIGMOD
9	8,228	SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB
10	526	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB