CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
Summary: CROSSBOW scales synchronous multi-GPU training without forcing large batches, using SMA: replicas independently descend while synchronously steering toward a global average trajectory. It packs and auto-tunes replicas per GPU, yielding 1.3–4× faster training than TensorFlow on eight GPUs. (summarized by gpt-5.6-luna on Jul 24 2026)
Incoming Non-self Citations Over Time
Authors
- 1. Alexandros Koliousis (Imperial College London)
- 2. Pijika Watcharapichat (Microsoft)
- 3. Matthias Weidlich (Humboldt University of Berlin)
- 4. Luo Mai (Imperial College London)
- 5. Paolo Costa (Microsoft)
- 6. Peter Pietzuch (Imperial College London)
BibTeX Citation
@article{koliousis_vldb19,
title = {{CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers}},
author = {Koliousis, Alexandros and Watcharapichat, Pijika and Weidlich, Matthias and Mai, Luo and Costa, Paolo and Pietzuch, Peter},
journal = {PVLDB},
series = {{VLDB} '19},
volume = {12},
number = {11},
pages = {1399--1413},
doi = {10.14778/3342263.3342276},
url = {https://doi.org/10.14778/3342263.3342276},
year = {2019}
}Incoming Citations (Sorted by Pagerank)
Showing 8 of 8 citing papers.
| Rank | Citing Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,152 | Cerebro: A Data System for Optimized Deep Learning Model Selection | 2020 | VLDB | 0.00011796404 |
| 2,287 | Pump Up the Volume: Processing Large Data on GPUs with Fast Interconnects | 2020 | SIGMOD | 8.691301e-05 |
| 3,055 | MG-Join: A Scalable Join for Massively Parallel Multi-GPU Architectures | 2021 | SIGMOD | 7.6983202e-05 |
| 4,097 | Distributed Deep Learning on Data Systems: A Comparative Analysis of Approaches | 2021 | VLDB | 6.8064316e-05 |
| 4,352 | Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity | 2024 | VLDB | 6.6396584e-05 |
| 9,006 | ANN Softmax: Acceleration of Extreme Classification Training | 2022 | VLDB | 5.2375792e-05 |
| 9,076 | TensorSocket: Shared Data Loading for Deep Learning Training | 2026 | SIGMOD | 5.2258409e-05 |
| 10,048 | The Image Calculator: 10x Faster Image-AI Inference by Replacing JPEG with Self-designing Storage Format | 2024 | SIGMOD | 5.0896901e-05 |
Previous
Page 1 / 1
Next
Outgoing Citations (Sorted by Pagerank)
Showing 4 of 4 cited papers.
Citations counted here include only citations to other VLDB/SIGMOD/CIDR/PODS papers in this database.
| Rank | Cited Paper | Year | Venue | Pagerank |
|---|---|---|---|---|
| 1,167 | DimmWitted: A Study of Main-Memory Statistical Analytics | 2014 | VLDB | 0.0001172597 |
| 2,186 | Heterogeneity-aware Distributed Parameter Servers | 2017 | SIGMOD | 8.8916253e-05 |
| 2,726 | FlexPS: Flexible Parallelism Control in Parameter Server Architecture | 2018 | VLDB | 8.0870349e-05 |
| 4,902 | Scalable Asynchronous Gradient Descent Optimization for Out-of-Core Models | 2017 | VLDB | 6.3617262e-05 |
Previous
Page 1 / 1
Next
Semantically Similar Papers
| # | Overall Rank | Paper | Year | Venue |
|---|---|---|---|---|
| 1 | 4,097 | Distributed Deep Learning on Data Systems: A Comparative Analysis of Approaches | 2021 | VLDB |
| 2 | 9,603 | Model-Parallel Model Selection for Deep Learning Systems | 2021 | SIGMOD |
| 3 | 4,965 | Heterogeneity-Aware Distributed Machine Learning Training via Partial Reduce | 2021 | SIGMOD |
| 4 | 3,450 | MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud | 2023 | VLDB |
| 5 | 523 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | 2020 | VLDB |
| 6 | 9,076 | TensorSocket: Shared Data Loading for Deep Learning Training | 2026 | SIGMOD |
| 7 | 9,907 | Scalable Graph Convolutional Network Training on Distributed-Memory Systems | 2023 | VLDB |
| 8 | 5,011 | Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism | 2023 | VLDB |
| 9 | 1,152 | Cerebro: A Data System for Optimized Deep Learning Model Selection | 2020 | VLDB |
| 10 | 3,812 | FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline | 2023 | VLDB |