FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
Summary: Dynamic expert management and placement address routing imbalance and dataflow fluctuation in sparse MoE training. A scheduling module monitors data flow and remaps hardware on the fly with a lightweight heuristic, boosting performance vs baselines. (summarized by gpt-5-nano on Feb 09 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Xiaonan Nie (Peking University)
- 2. Xupeng Miao (Carnegie Mellon University)
- 3. Zilong Wang (Microsoft)
- 4. Zichao Yang (Carnegie Mellon University)
- 5. Jilong Xue (Microsoft)
- 6. Lingxiao Ma (Microsoft)
- 7. Gang Cao (Beijing Academy of Artificial Intelligence)
- 8. Bin Cui (Peking University)
BibTeX Citation
@inproceedings{nie_sigmod23,
title = {{FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement}},
author = {Nie, Xiaonan and Miao, Xupeng and Wang, Zilong and Yang, Zichao and Xue, Jilong and Ma, Lingxiao and Cao, Gang and Cui, Bin},
series = {{SIGMOD} '23},
booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
publisher = {Association for Computing Machinery},
doi = {10.1145/3588964},
url = {https://dl.acm.org/doi/10.1145/3588964},
year = {2023}
}Incoming Citations (Sorted by Pagerank)
Showing 5 of 5 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 7,075 | Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity | 2024 | VLDB | 5.7099047e-05 |
| 8,034 | Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent | 2023 | VLDB | 5.502946e-05 |
| 8,101 | SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training | 2023 | VLDB | 5.4870581e-05 |
| 9,881 | The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format | 2024 | SIGMOD | 5.2040783e-05 |
| 10,769 | Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization | 2025 | SIGMOD | 5.093636e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 7 of 7 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 521 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB | 0.0001713368 |
| 2,485 | HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework | 2022 | VLDB | 8.5145736e-05 |
| 4,912 | HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training | 2022 | SIGMOD | 6.4481656e-05 |
| 5,003 | Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism | 2023 | VLDB | 6.4065691e-05 |
| 5,589 | An Experimental Evaluation of Large Scale GBDT Systems | 2019 | VLDB | 6.1559057e-05 |
| 8,034 | Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent | 2023 | VLDB | 5.502946e-05 |
| 10,114 | Towards Communication-efficient Vertical Federated Learning Training via Cache-enabled Local Updates | 2022 | VLDB | 5.1319012e-05 |
Previous
Page 1 / 1
Next