Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce

Summary: Partial-reduce: heterogeneity-aware all-reduce variant for distributed ML; decomposes synchronous all-reduce into parallel-asynchronous blocks to tolerate stragglers. Converges to a stationary point at sublinear SGD rate; adds dynamic, staleness-aware averaging and group-generation to avoid update isolation; prototype yields 1.21x–2x speedups. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 6116
Venue: SIGMOD
Year: 2021
Pagerank: 6.4290135e-05
Overall Rank: 4,956 | 66.00%
DOI: 10.1145/3448016.3452773

Incoming Non-self Citations Over Time

Authors

1. Xupeng Miao (Peking University; Tencent)
2. Xiaonan Nie (Peking University)
3. Yingxia Shao (Beijing Institute of Technology)
4. Zhi Yang (Peking University)
5. Jiawei Jiang (ETH Zurich)
6. Lingxiao Ma (Peking University)
7. Bin Cui (Peking University)

BibTeX Citation

@inproceedings{miao_sigmod21,
        title = {{Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce}},
        author = {Miao, Xupeng and Nie, Xiaonan and Shao, Yingxia and Yang, Zhi and Jiang, Jiawei and Ma, Lingxiao and Cui, Bin},
        series = {{SIGMOD} '21},
        booktitle = {Proceedings of the {ACM} {SIGMOD} International Conference on Management of Data},
        publisher = {Association for Computing Machinery},
        doi = {10.1145/3448016.3452773},
        url = {https://dl.acm.org/doi/10.1145/3448016.3452773},
        year = {2021}
}

Incoming Citations (Sorted by Pagerank)

Showing 14 of 14 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
1,132	SANCUS: Staleness-Aware Communication-Avoiding Full-Graph Decentralized Training in Large-Scale Graph Neural Networks	2022	VLDB	0.00012041292
2,485	HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework	2022	VLDB	8.5145736e-05
3,237	BlindFL: Vertical Federated Machine Learning without Peeking into Your Data	2022	SIGMOD	7.6089416e-05
4,912	HET-GMP: A Graph-based System Approach to Scaling Large Embedding Model Training	2022	SIGMOD	6.4481656e-05
5,003	Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism	2023	VLDB	6.4065691e-05
6,900	Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving	2025	SIGMOD	5.7430032e-05
7,075	Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity	2024	VLDB	5.7099047e-05
8,034	Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent	2023	VLDB	5.502946e-05
8,101	SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	5.4870581e-05
8,321	FusionFlow: Accelerating Data Preprocessing for Machine Learning with CPU-GPU Cooperation	2024	VLDB	5.4544352e-05
9,475	BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach	2023	SIGMOD	5.2634238e-05
9,881	The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format	2024	SIGMOD	5.2040783e-05
10,596	NeutronCloud: Resource-Aware Distributed GNN Training in Fluctuating Cloud Environments	2026	VLDB	5.093636e-05
10,849	SimRN: Trajectory Similarity Learning in Road Networks based on Distributed Deep Reinforcement Learning	2025	VLDB	5.093636e-05

Outgoing Citations (Sorted by Pagerank)

Showing 6 of 6 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
521	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.0001713368
1,150	DimmWitted: A Study of Main-Memory Statistical Analytics	2014	VLDB	0.00011943462
2,162	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	9.0581831e-05
2,347	Vertica-ML: Distributed Machine Learning in Vertica Database	2020	SIGMOD	8.7157552e-05
4,083	SketchML: Accelerating Distributed Machine Learning with Data Sketches	2018	SIGMOD	6.9160949e-05
5,345	PS2: Parameter Server on Spark	2019	SIGMOD	6.2586047e-05

Semantically Similar Papers

#	Overall Rank	Paper	Year	Venue
1	5,876	BAGUA: Scaling up Distributed Learning with System Relaxations	2022	VLDB
2	1,079	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB
3	9,371	Towards an Optimized GROUP BY Abstraction for Large-Scale Machine Learning	2021	VLDB
4	9,729	Scalable Graph Convolutional Network Training on Distributed-Memory Systems	2023	VLDB
5	6,519	Dynamic Parameter Allocation in Parameter Servers	2020	VLDB
6	10,113	Distributed Learning of Fully Connected Neural Networks using Independent Subnet Training	2022	VLDB
7	521	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB
8	4,799	Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models	2017	VLDB
9	8,101	SDP_PIPE: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB
10	2,162	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD