Magnus: A Holistic Approach to Data Management for Large-Scale Machine Learning Workloads

Summary: Magnus is a holistic data-management layer on Apache Iceberg tailored to large-scale ML workloads (wide tables, multimodal) combining resource-efficient storage formats with built-in vector/inverted indexes to speed retrieval. It adds scalable Git-like metadata branching, lightweight merge-on-read upsert, and native LRM/LMM training support; deployed at ByteDance with substantial real-world gains. (summarized by gpt-5-mini on Feb 09 2026)

Paper ID: 14099
Venue: VLDB
Year: 2025
Pagerank: 4.1945683e-05
Overall Rank: 10,777 | 25.03%
DOI: 10.14778/3750601.3750620

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank

Outgoing Citations (Sorted by Pagerank)

Showing 14 of 14 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
746	Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	2020	VLDB	0.00017326979
1,377	Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics	2021	CIDR	0.00012296941
2,062	Dremel: A Decade of Interactive SQL Analysis at Web Scale	2020	VLDB	9.6481955e-05
2,473	Photon: A Fast Query Engine for Lakehouse Systems	2022	SIGMOD	8.7237281e-05
4,530	Big Metadata: When Metadata is Big Data	2021	VLDB	6.1075429e-05
5,318	Analyzing and Comparing Lakehouse Storage Systems	2023	CIDR	5.5715872e-05
6,402	BigLake: BigQuery’s Evolution toward a Multi-Cloud Lakehouse	2024	SIGMOD	5.079818e-05
6,715	Shared Foundations: Modernizing Meta's Data Lakehouse	2023	CIDR	4.9509939e-05
7,469	Bullion: A Column Store for Machine Learning	2025	CIDR	4.7204398e-05
7,492	Krypton: Real-time Serving and Analytical SQL Engine at ByteDance	2023	VLDB	4.7180617e-05
7,814	Deep Lake: a Lakehouse for Deep Learning	2023	CIDR	4.6439001e-05
7,907	Petabyte-Scale Row-Level Operations in Data Lakehouses	2024	VLDB	4.6205839e-05
9,155	Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance	2024	VLDB	4.3849295e-05
9,236	The Hopsworks Feature Store for Machine Learning	2024	SIGMOD	4.3690661e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
8,637	Machine Learning for Data Management: Problems and Solutions	2018	SIGMOD	4.479892e-05
7,020	LLM for Data Management	2024	VLDB	4.8595728e-05
7,375	BG3: A Cost Effective and I/O Efficient Graph Database in ByteDance	2024	SIGMOD	4.7491278e-05
4,549	Database-Agnostic Workload Management	2019	CIDR	6.0926728e-05
495	Milvus: A Purpose-Built Vector Data Management System	2021	SIGMOD	0.00021767688
13,171	Reimagining Deep Learning Systems Through the Lens of Data Systems	2024	VLDB	-
9,236	The Hopsworks Feature Store for Machine Learning	2024	SIGMOD	4.3690661e-05
7,411	ItemSuggest: A Data Management Platform for Machine Learned Ranking Services	2019	CIDR	4.7364436e-05
4,003	Data Platform for Machine Learning	2019	SIGMOD	6.54347e-05
1,532	Data Management in Machine Learning: Challenges, Techniques, and Systems	2017	SIGMOD	0.00011472681