pytorch-multigpu
Multi GPU Training Code for Deep Learning with PyTorch. Train PyramidNet for CIFAR10 classification task. This code is for comparing several ways of multi-GPU training.
Requirement
- Python 3
- PyTorch 1.0.0+
- TorchVision
- TensorboardX
Usage
single gpu
cd single_gpu
python train.py
DataParallel
cd data_parallel
python train.py --gpu_devices 0 1 2 3 --batch_size 768
DistributedDataParallel
cd dist_parallel
python train.py --gpu_device 0 1 2 3 --batch_size 768
Performance
single gpu
- batch size: 240
- batch time: 6s
- training time: 22 min
- gpu util: 99 %
- gpu memory: 10 G
DataParallel(4 k80)
- batch size: 768
- batch time: 5s
- training time: 5 min
- gpu util: 99 %
- gpu memory: 10 G * 4