Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization

Summary: Malleus enables straggler-resilient hybrid training via per-GPU profiling and a planning algorithm that optimizes GPU groups, pipelines, layers, and data. It re-plans and migrates state on the fly to sustain stability, operating under dynamic straggler distributions and delivering 2.63–5.28x efficiency on LLMs up to 110B. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 7303
Venue: SIGMOD
Year: 2025
Pagerank: 5.093636e-05
Overall Rank: 10,769 | 26.12%
DOI: 10.1145/3725322

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

1. Haoyang Li (Peking University)
2. Fangcheng Fu (Peking University)
3. Hao Ge (Peking University)
4. Sheng Lin (Peking University)
5. Xuanyu Wang (Peking University)
6. Jiawen Niu (Peking University)
7. Yujie Wang (Peking University)
8. Hailin Zhang (Peking University)
9. Xiaonan Nie (Peking University)
10. Bin Cui (Peking University)

BibTeX Citation

@inproceedings{li_sigmod25,
        title = {{Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization}},
        author = {Li, Haoyang and Fu, Fangcheng and Ge, Hao and Lin, Sheng and Wang, Xuanyu and Niu, Jiawen and Wang, Yujie and Zhang, Hailin and Nie, Xiaonan and Cui, Bin},
        series = {{SIGMOD} '25},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3725322},
        url = {https://dl.acm.org/doi/10.1145/3725322},
        year = {2025}
}

Incoming Citations (Sorted by Pagerank)

Showing 1 of 1 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
10,380	Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment	2026	SIGMOD	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 18 of 18 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
2,162	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	9.0581831e-05
2,298	GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization	2024	VLDB	8.7886538e-05
2,473	PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel	2023	VLDB	8.5326287e-05
2,852	The Dawn of Natural Language to SQL: Are We Fully Ready?	2024	VLDB	8.0455088e-05
3,541	Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines	2022	SIGMOD	7.3280673e-05
4,083	SketchML: Accelerating Distributed Machine Learning with Data Sketches	2018	SIGMOD	6.9160949e-05
4,515	ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models	2024	VLDB	6.6492389e-05
5,003	Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism	2023	VLDB	6.4065691e-05
5,282	GoldMiner: Elastic Scaling of Training Data Pre-Processing Pipelines for Deep Learning	2023	SIGMOD	6.2840712e-05
5,876	BAGUA: Scaling up Distributed Learning with System Relaxations	2022	VLDB	6.0557672e-05
7,075	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB	5.7099047e-05
7,102	Data and AI Model Markets: Opportunities for Data and Model Sharing, Discovery, and Integration	2023	VLDB	5.7030531e-05
7,232	Saga: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Applications	2023	SIGMOD	5.6659017e-05
7,847	Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines	2024	VLDB	5.5330423e-05
8,034	Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent	2023	VLDB	5.502946e-05
8,883	FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement	2023	SIGMOD	5.351513e-05
9,469	How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study	2024	VLDB	5.2634238e-05
9,475	BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach	2023	SIGMOD	5.2634238e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	521	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB
2	2,473	PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel	2023	VLDB
3	8,725	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB
4	5,305	Parallel Training of Knowledge Graph Embedding Models: A Comparison of Techniques	2022	VLDB
5	1,079	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB
6	5,003	Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism	2023	VLDB
7	9,414	Model-Parallel Model Selection for Deep Learning Systems	2021	SIGMOD
8	4,956	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD
9	10,219	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD
10	10,380	Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment	2026	SIGMOD