MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

Summary: MiCS minimizes communication by shrinking collective participant sets to exploit heterogeneous cloud bandwidth, avoid slow links, and amortize global gradient synchronization. On AWS it achieves up to 2.89x throughput vs prior systems and near-linear weak scaling (99.4% at 512 GPUs for a 100B model), improving GPU utilization over DeepSpeed in constrained public-cloud networks. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 13199
Venue: VLDB
Year: 2023
Pagerank: 8.9766205e-05
Overall Rank: 2,352 | 83.64%
DOI: 10.14778/3561261.3561265

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 5 of 5 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
2,902	PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel	2023	VLDB	7.93939e-05
3,995	How Large Language Models Will Disrupt Data Management	2023	VLDB	6.5513237e-05
7,152	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB	4.8154191e-05
9,805	MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training	2025	SIGMOD	4.2805224e-05
10,626	LobRA: Multi-tenant Fine-tuning over Heterogeneous Data	2025	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 0 of 0 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
10,298	NeutronCloud: Resource-Aware Distributed GNN Training in Fluctuating Cloud Environments	2026	VLDB	4.1945683e-05
5,333	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD	5.5656575e-05
8,126	SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	4.5796615e-05
4,180	FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline	2023	VLDB	6.3793352e-05
10,285	Resilience-Aware Elastic Scaling for Cloud-Native Online DL Training on Multi-Tenant GPU Clusters	2026	VLDB	4.1945683e-05
3,087	Scalable and Efficient Full-Graph GNN Training for Large Graphs	2023	SIGMOD	7.5939896e-05
1,103	Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture	2021	VLDB	0.00014025101
3,363	CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers	2019	VLDB	7.1731921e-05
9,596	Scalable Graph Convolutional Network Training on Distributed-Memory Systems	2023	VLDB	4.319218e-05
9,319	How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study	2024	VLDB	4.3556432e-05