Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads
Summary: Magnus is a holistic data-management layer on Apache Iceberg tailored to large-scale ML workloads (wide tables, multimodal) combining resource-efficient storage formats with built-in vector/inverted indexes to speed retrieval. It adds scalable Git-like metadata branching, lightweight merge-on-read upsert, and native LRM/LMM training support; deployed at ByteDance with substantial real-world gains. (summarized by gpt-5-mini on Feb 09 2026)
Incoming Non-self Citations Over Time
No non-self incoming citations found for this paper in this database.
Authors
- 1. Jun Song (ByteDance)
- 2. Jingyi Ding (Zhejiang University)
- 3. Irshad Kandy (ByteDance)
- 4. Yanghao Lin (ByteDance)
- 5. Zhongjia Wei (ByteDance)
- 6. Zilong Zhou (ByteDance)
- 7. Zhiwei Peng (ByteDance)
- 8. Jixi Shan (ByteDance)
- 9. Hongyue Mao (ByteDance)
- 10. Xiuqi Huang (Zhejiang University)
- 11. Xun Song (ByteDance)
- 12. Cheng Chen (ByteDance)
- 13. Yanjia Li (ByteDance)
- 14. Tianhao Yang (ByteDance)
- 15. Wei Jia (ByteDance)
- 16. Xiaohong Dong (ByteDance)
- 17. Kang Lei (ByteDance)
- 18. Rui Shi (ByteDance)
- 19. Pengwei Zhao (ByteDance)
- 20. Wei Chen (Zhejiang University)
BibTeX Citation
@article{song_vldb25,
title = {{Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads}},
author = {Song, Jun and Ding, Jingyi and Kandy, Irshad and Lin, Yanghao and Wei, Zhongjia and Zhou, Zilong and Peng, Zhiwei and Shan, Jixi and Mao, Hongyue and Huang, Xiuqi and Song, Xun and Chen, Cheng and Li, Yanjia and Yang, Tianhao and Jia, Wei and Dong, Xiaohong and Lei, Kang and Shi, Rui and Zhao, Pengwei and Chen, Wei},
journal = {PVLDB},
series = {{VLDB} '25},
volume = {18},
number = {12},
pages = {4964--4977},
doi = {10.14778/3750601.3750620},
url = {https://doi.org/10.14778/3750601.3750620},
year = {2025}
}
Incoming Citations (Sorted by Pagerank)
Showing 0 of 0 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 14 of 14 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 8,676 | Machine Learning for Data Management: Problems and Solutions | 2018 | SIGMOD |
| 2 | 6,101 | LLM for Data Management | 2024 | VLDB |
| 3 | 6,521 | BG3: A Cost Effective and I/O Efficient Graph Database in ByteDance | 2024 | SIGMOD |
| 4 | 4,053 | Database-Agnostic Workload Management | 2019 | CIDR |
| 5 | 286 | Milvus: A Purpose-Built Vector Data Management System | 2021 | SIGMOD |
| 6 | 13,375 | Reimagining Deep Learning Systems Through the Lens of Data Systems | 2024 | VLDB |
| 7 | 9,385 | The Hopsworks Feature Store for Machine Learning | 2024 | SIGMOD |
| 8 | 7,855 | ItemSuggest: A Data Management Platform for Machine Learned Ranking Services | 2019 | CIDR |
| 9 | 4,592 | Data Platform for Machine Learning | 2019 | SIGMOD |
| 10 | 1,250 | Data Management in Machine Learning: Challenges, Techniques, and Systems | 2017 | SIGMOD |