Benchmark on Deep Learning Frameworks and GPUs

Performance of popular deep learning frameworks and GPUs are compared, including the effect of adjusting the floating point precision (the new Volta architecture allows performance boost by utilizing half/mixed-precision calculations.)

Deep Learning Frameworks

Note: Docker images available from NVIDIA GPU Cloud were used so as to make benchmarking controlled and repeatable by anyone.

• PyTorch 0.3.0

  • docker pull nvcr.io/nvidia/pytorch:17.12

• PyTorch 1.0.0 (CUDA 10.0, cuDNN 7.4.2)

  • docker pull nvcr.io/nvidia/pytorch:19.01-py3 (note: requires login API key to NGC registry)

• Caffe2 0.8.1

  • docker pull nvcr.io/nvidia/caffe2:17.12

• TensorFlow 1.4.0 (note: this is TensorFlow 1.4.0 compiled against CUDA 9 and CuDNN 7)

  • docker pull nvcr.io/nvidia/tensorflow:17.12

• TensorFlow 1.5.0

• TensorFlow 1.12.0 (CUDA 10.0, cuDNN 7.4.2)

  • docker pull nvcr.io/nvidia/tensorflow:19.01-py3 (note: requires login API key to NGC registry)

• MXNet 1.0.0 (anyone interested?)

  • docker pull nvcr.io/nvidia/mxnet:17.12

• CNTK (anyone interested?)

  • docker pull nvcr.io/nvidia/cntk:17.12

GPUs

*: F16 (single precision) TFLOPS on TensorCores.

CUDA / CuDNN

• CUDA 9.0.176
• CuDNN 7.0.0.5
• NVIDIA driver 387.34.

Except where noted.

Networks

• VGG16
• Resnet152
• Densenet161
• Any others you might be interested in?

Benchmark Results

PyTorch 0.3.0

The results are based on running the models with images of size 224 x 224 x 3 with a batch size of 16. "Eval" shows the duration for a single forward pass averaged over 20 passes. "Train" shows the duration for a pair of forward and backward passes averaged over 20 runs. In both scenarios, 20 runs of warm up is performed and those are not counted towards the measured numbers.

Titan V gets a significant speed up when going to half precision by utilizing its Tensor cores, while 1080 Ti gets a small speed up with half precision computation. Similarly, the numbers from V100 on an Amazon p3 instance is shown. It is faster than Titan V and the speed up when going to half-precision is similar to that of Titan V.

32-bit

16-bit

PyTorch 1.0.0

32-bit

16-bit

Tensorflow 1.4.0

32-bit

16-bit

TensorFlow 1.5.0

32-bit

16-bit

TensorFlow 1.12.0

32-bit

16-bit

Caffe2 0.8.1

32-bit

16-bit

Comparison Graphs

Comparison of Titan V vs 1080 Ti, PyTorch 0.3.0 vs Tensorflow 1.4.0 vs Caffe2 0.8.1, and FP32 vs FP16 in terms of images processed per second:

Contributors

• Yusaku Sako
• Bartosz Ludwiczuk (thank you for supplying the V100 numbers)