PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Summary: Introduces PyTorch Fully Sharded Data Parallel (FSDP), an industry-grade, non-intrusive sharding framework co-designed with PyTorch internals (Tensor, dispatcher, CUDA allocator) to enable training of much larger models than DDP. FSDP bundles memory and communication optimizations across hardware to achieve near-linear TFLOPS scalability and DDP-comparable throughput while drastically reducing memory footprint. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13212
Venue: VLDB
Year: 2023
Pagerank: 7.93939e-05
Overall Rank: 2,902 | 79.82%
DOI: 10.14778/3611540.3611569

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 11 of 11 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
3,995	How Large Language Models Will Disrupt Data Management	2023	VLDB	6.5513237e-05
8,520	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB	4.4937074e-05
9,805	MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training	2025	SIGMOD	4.2805224e-05
10,089	Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment	2026	SIGMOD	4.1945683e-05
10,183	Mixtera: A Data Plane for Foundation Model Training	2026	SIGMOD	4.1945683e-05
10,398	Sequoia: An Accessible and Extensible Framework for Privacy-Preserving Machine Learning over Distributed Data	2025	SIGMOD	4.1945683e-05
10,492	Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization	2025	SIGMOD	4.1945683e-05
10,580	GPEmu: A GPU Emulator for Faster and Cheaper Prototyping and Evaluation of Deep Learning System Research	2025	VLDB	4.1945683e-05
10,626	LobRA: Multi-tenant Fine-tuning over Heterogeneous Data	2025	VLDB	4.1945683e-05
10,693	Evoschema: Towards Text-To-Sql Robustness Against Schema Evolution	2025	VLDB	4.1945683e-05
13,122	DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems	2025	VLDB	-

Outgoing Citations (Sorted by Pagerank)

Showing 2 of 2 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921
2,352	MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud	2023	VLDB	8.9766205e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
9,965	Distributed Learning of Fully Connected Neural Networks using Independent Subnet Training	2022	VLDB	4.2269436e-05
4,395	Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models	2017	VLDB	6.2244283e-05
1,402	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB	0.00012180605
3,087	Scalable and Efficient Full-Graph GNN Training for Large Graphs	2023	SIGMOD	7.5939896e-05
4,180	FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline	2023	VLDB	6.3793352e-05
5,333	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD	5.5656575e-05
9,264	Model-Parallel Model Selection for Deep Learning Systems	2021	SIGMOD	4.3675421e-05
8,735	TensorSocket: Shared Data Loading for Deep Learning Training	2026	SIGMOD	4.456315e-05
8,126	SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	4.5796615e-05
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921