Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
Summary: Mitigates data-induced imbalances in Transformer training—uneven sequence-length sampling and packing mismatch between attention time (quadratic) and memory (linear)—by jointly optimizing parallel strategy and data assignment. Hydraulis applies dynamic heterogeneous parallelism and a two-stage data assignment to balance intra- and inter-replica workloads, boosting throughput 1.32–2.66×. (summarized by gpt-5-mini on Feb 11 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Haoyang Li
- 2. Fangcheng Fu
- 3. Sheng Lin
- 4. Hao Ge
- 5. Xuanyu Wang
- 6. Jiawen Niu
- 7. Jinbao Xue
- 8. Yangyu Tao
- 9. Di Wang
- 10. Jie Jiang
- 11. Bin Cui
Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 6 of 6 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 411 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB | 0.00023906921 |
| 2,902 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | 2023 | VLDB | 7.93939e-05 |
| 6,377 | Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism | 2023 | VLDB | 5.0911095e-05 |
| 9,805 | MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training | 2025 | SIGMOD | 4.2805224e-05 |
| 10,492 | Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization | 2025 | SIGMOD | 4.1945683e-05 |
| 10,626 | LobRA: Multi-tenant Fine-tuning over Heterogeneous Data | 2025 | VLDB | 4.1945683e-05 |
Previous
Page 1 / 1
Next