BAGUA: Scaling up Distributed Learning with System Relaxations

Summary: MPI-style, modular BAGUA provides system-relaxation primitives (quantization, decentralization, delayed communication) for distributed data-parallel training. Enables rapid prototyping of advanced distributed-learning algorithms; delivers up to 2x end-to-end speedups over PyTorch-DDP/Horovod/BytePS on 128 GPUs and analyzes performance tradeoffs across network conditions. (summarized by gpt-5-nano on Feb 09 2026)

Paper ID: 12951
Venue: VLDB
Year: 2022
Pagerank: 5.3527734e-05
Overall Rank: 5,720 | 60.21%
DOI: 10.14778/3503585.3503590

Incoming Non-self Citations Over Time

Authors

Incoming Citations (Sorted by Pagerank)

Showing 4 of 4 citing papers.

Rank	Citing Paper	Year	Venue	Pagerank
7,306	DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines	2022	CIDR	4.7678574e-05
10,398	Sequoia: An Accessible and Extensible Framework for Privacy-Preserving Machine Learning over Distributed Data	2025	SIGMOD	4.1945683e-05
10,492	Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization	2025	SIGMOD	4.1945683e-05
10,626	LobRA: Multi-tenant Fine-tuning over Heterogeneous Data	2025	VLDB	4.1945683e-05

Outgoing Citations (Sorted by Pagerank)

Showing 12 of 12 cited papers.

Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.

Rank	Cited Paper	Year	Venue	Pagerank
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921
557	SystemML: Declarative Machine Learning on Spark	2016	VLDB	0.00020197988
1,402	Hybrid Parallelization Strategies for Large-Scale Machine Learning in SystemML	2014	VLDB	0.00012180605
1,942	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	0.00010012691
2,440	FlexPS: Flexible Parallelism Control in Parameter Server Architecture	2018	VLDB	8.8119143e-05
2,642	Vertica-ML: Distributed Machine Learning in Vertica Database	2020	SIGMOD	8.3851878e-05
3,099	DB4ML – An In-Memory Database Kernel with Machine Learning Support	2020	SIGMOD	7.5642871e-05
4,263	Mariana: Tencent Deep Learning Platform and its Applications	2014	VLDB	6.3106444e-05
4,964	PS2: Parameter Server on Spark	2019	SIGMOD	5.7965988e-05
5,821	Tensor Relational Algebra for Distributed Machine Learning System Design	2021	VLDB	5.3134851e-05
6,811	In-database Distributed Machine Learning: Demonstration using Teradata SQL Engine	2019	VLDB	4.9200998e-05
9,469	DimBoost: Boosting Gradient Boosting Decision Tree to Higher Dimensions	2018	SIGMOD	4.3342363e-05

Semantically Similar Papers

Overall Rank	Paper	Year	Venue	Pagerank
3,363	CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers	2019	VLDB	7.1731921e-05
6,471	Dynamic Parameter Allocation in Parameter Servers	2020	VLDB	5.0511668e-05
8,126	SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training	2023	VLDB	4.5796615e-05
543	MLbase: A Distributed Machine-learning System	2013	CIDR	0.00020526854
5,821	Tensor Relational Algebra for Distributed Machine Learning System Design	2021	VLDB	5.3134851e-05
9,222	Towards an Optimized GROUP BY Abstraction for Large-Scale Machine Learning	2021	VLDB	4.3698672e-05
2,400	ByteGNN: Efficient Graph Neural Network Training at Large Scale	2022	VLDB	8.8955105e-05
1,942	Heterogeneity-aware Distributed Parameter Servers	2017	SIGMOD	0.00010012691
411	PyTorch Distributed: Experiences on Accelerating Data Parallel Training	2020	VLDB	0.00023906921
5,333	Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce	2021	SIGMOD	5.5656575e-05