ZenFlow: Enabling Stall-Free Offloading for LLM Training

Summary: ZenFlow avoids GPU stalls in CPU-offloaded LLM fine-tuning by updating important gradients in place on GPU while asynchronously accumulating less important ones on CPU. It exploits stable spatio-temporal gradient locality for lightweight prioritization, achieving up to 5× speedup and 2× lower PCIe traffic without accuracy loss. (summarized by gpt-6-luna on Oct 22 2026)

Paper ID: ha3b0115c18abc324
Venue: SIGMOD
Year: 2027
Pagerank: 4.9605343e-05
Overall Rank: 10,415 | 30.19%
DOI: 10.1145/3837130
PDF: Download (CC BY 4.0)

Incoming Non-self Citations Over Time

No non-self incoming citations found for this paper in this database.

Authors

1. Tingfeng Lan (University of Virginia)
2. Yusen Wu (University of Virginia)
3. Bin Ma (University of California Merced)
4. Zhaoyuan Su (University of Virginia)
5. Rui Yang (University of Virginia)
6. Tekin Bicer (Argonne National Laboratory)
7. Masahiro Tanaka (AnyScale)
8. Olatunji Ruwase (Snowflake)
9. Dong Li (University of California Merced)
10. Yue Cheng (University of Virginia)

BibTeX Citation

@inproceedings{lan_sigmod27,
        title = {{ZenFlow: Enabling Stall-Free Offloading for LLM Training}},
        author = {Lan, Tingfeng and Wu, Yusen and Ma, Bin and Su, Zhaoyuan and Yang, Rui and Bicer, Tekin and Tanaka, Masahiro and Ruwase, Olatunji and Li, Dong and Cheng, Yue},
        series = {{SIGMOD} '27},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3837130},
        url = {https://dl.acm.org/doi/10.1145/3837130},
        year = {2027}
}

Incoming Citations (Sorted by Pagerank)

Showing 0 of 0 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank

Outgoing Citations (Sorted by Pagerank)

Showing 3 of 3 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
1,612	PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel	2023	VLDB	0.00010061487
7,120	DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud	2024	VLDB	5.580601e-05
8,916	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB	5.2361195e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	10,828	Mil: Cost-guided Minimum Makespan Scheduling for Applications of Multiple LLMs	2026	VLDB
2	11,257	Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization	2025	SIGMOD
3	4,360	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB
4	10,016	MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training	2025	SIGMOD
5	8,916	mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs	2025	VLDB
6	10,909	Unified Static–Dynamic Pruning for Efficient LLM Inference	2026	VLDB
7	13,628	OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration	2026	VLDB
8	13,625	Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking	2026	SIGMOD
9	3,800	FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline	2023	VLDB
10	10,513	DFLOP: A Data-driven Framework for Multimodal LLM Training Pipeline Optimization	2026	SIGMOD