Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers

Summary: Harmony rethinks GPU memory management and data movement to train massive DNNs on a single commodity server. Redesigned scheduling and CPU–GPU data paths cut swap by up to 100x and yield up to 7.6x throughput over optimized virtual memory baselines. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12949
Venue: VLDB
Year: 2022
Pagerank: 5.3954887e-05
Overall Rank: 8,637 | 40.75%
DOI: 10.14778/3551793.3551828

Incoming Non-self Citations Over Time

Authors

1. Youjie Li (University of Illinois Urbana-Champaign)
2. Amar Phanishayee (Microsoft)
3. Derek Murray (Lacework)
4. Jakub Tarnawski (Microsoft)
5. Nam Sung Kim (University of Illinois Urbana-Champaign)

BibTeX Citation

@article{li_vldb22,
        title = {{Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers}},
        author = {Li, Youjie and Phanishayee, Amar and Murray, Derek and Tarnawski, Jakub and Kim, Nam Sung},
        journal = {PVLDB},
        series = {{VLDB} '22},
        volume = {15},
        number = {11},
        pages = {2747--2760},
        doi = {10.14778/3551793.3551828},
        url = {https://doi.org/10.14778/3551793.3551828},
        year = {2022}
}

Incoming Citations (Sorted by Pagerank)

Showing 2 of 2 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
7,075	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB	5.7099047e-05
9,475	BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach	2023	SIGMOD	5.2634238e-05

Outgoing Citations (Sorted by Pagerank)

Showing 1 of 1 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
521	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.0001713368

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	8,883	FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement	2023	SIGMOD
2	5,760	DynaHB: A Communication-Avoiding Asynchronous Distributed Framework with Hybrid Batches for Dynamic GNN Training	2024	VLDB
3	10,907	Effective and Efficient Distributed Temporal Graph Learning through Hotspot Memory Sharing	2025	VLDB
4	9,414	Model-Parallel Model Selection for Deep Learning Systems	2021	SIGMOD
5	1,234	Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture	2021	VLDB
6	8,905	TensorSocket: Shared Data Loading for Deep Learning Training	2026	SIGMOD
7	9,292	COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression	2022	VLDB
8	9,729	Scalable Graph Convolutional Network Training on Distributed-Memory Systems	2023	VLDB
9	9,219	MemFlow: Memory-Aware Distributed Deep Learning	2020	SIGMOD
10	10,034	HARMONY: A Scalable Distributed Vector Database for High-Throughput Approximate Nearest Neighbor Search	2026	SIGMOD