mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
Summary: mLoRA: LoRA-aware pipeline parallelism and a LoRA-efficient operator to parallelize multiple LoRA adapter fine-tuning across GPUs/machines, reducing communication and improving GPU utilization. Cuts average fine-tuning time ~30% vs FSDP and enables larger models on fewer GPUs. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Zhengmao Ye (Sichuan University)
- 2. Dengchun Li (Sichuan University)
- 3. Zetao Hu (Sichuan University)
- 4. Tingfeng Lan (Sichuan University)
- 5. Jian Sha (Ant Financial)
- 6. Shicong Zhang (Zhejiang New Internet Exchange Center)
- 7. Lei Duan (Sichuan University)
- 8. Jie Zuo (Sichuan University)
- 9. Hui Lu (University of Texas)
- 10. Yuanchun Zhou (Chinese Academy of Sciences)
- 11. Mingjie Tang (Sichuan University)
BibTeX Citation
@article{ye_vldb25,
title = {{mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs}},
author = {Ye, Zhengmao and Li, Dengchun and Hu, Zetao and Lan, Tingfeng and Sha, Jian and Zhang, Shicong and Duan, Lei and Zuo, Jie and Lu, Hui and Zhou, Yuanchun and Tang, Mingjie},
journal = {PVLDB},
series = {{VLDB} '25},
volume = {18},
number = {6},
pages = {1948--1961},
doi = {10.14778/3725688.3725718},
url = {https://doi.org/10.14778/3725688.3725718},
year = {2025}
}Incoming Citations (Sorted by Pagerank)
Showing 1 of 1 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 10,880 | LobRA: Multi-tenant Fine-tuning over Heterogeneous Data | 2025 | VLDB | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 7 of 7 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 279 | Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation | 2024 | VLDB | 0.00022468369 |
| 521 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB | 0.0001713368 |
| 713 | Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes | 2024 | VLDB | 0.00014672521 |
| 2,298 | GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian Optimization | 2024 | VLDB | 8.7886538e-05 |
| 2,473 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | VLDB | 8.5326287e-05 |
| 4,515 | ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models | 2024 | VLDB | 6.6492389e-05 |
| 7,075 | Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity | 2024 | VLDB | 5.7099047e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 9,414 | Model-Parallel Model Selection for Deep Learning Systems | 2021 | SIGMOD |
| 2 | 10,410 | TranSQL+: Serving Large Language Models with SQL on Low-Resource Hardware | 2026 | SIGMOD |
| 3 | 13,288 | Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking | 2026 | SIGMOD |
| 4 | 2,473 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | VLDB |
| 5 | 10,769 | Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization | 2025 | SIGMOD |
| 6 | 13,343 | Database Perspective on LLM Inference Systems | 2025 | VLDB |
| 7 | 10,614 | LLM-AutoDP: Automatic Data Processing via LLM Agents for Model Fine-tuning | 2026 | VLDB |
| 8 | 9,880 | MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training | 2025 | SIGMOD |
| 9 | 10,219 | DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization | 2026 | SIGMOD |
| 10 | 10,880 | LobRA: Multi-tenant Fine-tuning over Heterogeneous Data | 2025 | VLDB |