Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce

Summary: Partial-reduce: heterogeneity-aware all-reduce variant for distributed ML; decomposes synchronous all-reduce into parallel-asynchronous blocks to tolerate stragglers. Converges to a stationary point at sublinear SGD rate; adds dynamic, staleness-aware averaging and group-generation to avoid update isolation; prototype yields 1.21x–2x speedups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6054
Venue: SIGMOD
Year: 2021
Pagerank: 5.5656575e-05
Overall Rank: 5,333 | 62.91%
DOI: 10.1145/3448016.3452773

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 14 of 14 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,160	Sancus: Staleness-Aware Communication-Avoiding Full-Graph Decentralized Training in Large-Scale Graph Neural Networks	2022	VLDB	0.00013586221
2,677	HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework	2022	VLDB	8.3268401e-05
3,506	BlindFL: Vertical Federated Machine Learning without Peeking into Your Data	2022	SIGMOD	7.0291192e-05
5,052	HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training	2022	SIGMOD	5.7337977e-05
6,377	Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism	2023	VLDB	5.0911095e-05
7,152	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB	4.8154191e-05
7,536	Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent	2023	VLDB	4.7176331e-05
8,126	SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	4.5796615e-05
8,348	FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation	2024	VLDB	4.5410024e-05
9,326	BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach	2023	SIGMOD	4.3556432e-05
9,677	Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving	2025	SIGMOD	4.3047774e-05
9,806	The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format	2024	SIGMOD	4.2805224e-05
10,298	NeutronCloud: Resource-Aware Distributed GNN Training in Fluctuating Cloud Environments	2026	VLDB	4.1945683e-05
10,588	SimRN: Trajectory Similarity Learning in Road Networks based on Distributed Deep Reinforcement Learning	2025	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 6 of 6 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921
1,044	DimmWitted: A Study of Main-Memory Statistical Analytics	2014	VLDB	0.00014475229
1,942	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	0.00010012691
2,642	Vertica-ML: Distributed Machine Learning in Vertica Database	2020	SIGMOD	8.3851878e-05
3,808	SketchML: Accelerating Distributed Machine Learning with Data Sketches	2018	SIGMOD	6.7455428e-05
4,964	PS2: Parameter Server on Spark	2019	SIGMOD	5.7965988e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
10,638	Heta: Distributed Training of Heterogeneous Graph Neural Networks	2025	VLDB	4.1945683e-05
5,720	BAGUA: Scaling up Distributed Learning with System Relaxations	2022	VLDB	5.3527734e-05
9,222	Towards an Optimized GROUP BY Abstraction for Large-Scale Machine Learning	2021	VLDB	4.3698672e-05
9,596	Scalable Graph Convolutional Network Training on Distributed-Memory Systems	2023	VLDB	4.319218e-05
6,471	Dynamic Parameter Allocation in Parameter Servers	2020	VLDB	5.0511668e-05
9,965	Distributed Learning of Fully Connected Neural Networks using Independent Subnet Training	2022	VLDB	4.2269436e-05
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921
4,395	Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models	2017	VLDB	6.2244283e-05
8,126	SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	4.5796615e-05
1,942	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	0.00010012691