songhan/DSD

• March 15, 2019: for our most updated work on model compression and acceleration, please reference:

  ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (ICLR’19)

  AMC: AutoML for Model Compression and Acceleration on Mobile Devices (ECCV’18)

  HAQ: Hardware-Aware Automated Quantization (CVPR’19)

  Defenstive Quantization: When Efficiency Meets Robustness (ICLR'19)

DSD Model Zoo

This repo contains pre-trained models by Dense-Sparse-Dense(DSD) training on Imagenet.

Compared to conventional training method, dense→sparse→dense (DSD) training yielded higher accuracy with same model architecture.

Sparsity is a powerful form of regularization. Our intuition is that, once the network arrives at a local minimum given the sparsity constraint, relaxing the constraint gives the network more freedom to escape the saddle point and arrive at a higher-accuracy local minimum.

Feel free to use the better-accuracy DSD models to help your research. If you find DSD traing useful, please cite the following paper:

@article{han2016_DSD,
  title={DSD: Dense-Sparse-Dense Training for Deep Neural Networks},
  author={Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, William J. Dally},
  journal={International Conference on Learning Representations (ICLR)},
  year={2017}
}

Download:

AlexNet_DSD

VGG16_DSD

GoogleNet_DSD

SqueezeNet_DSD

ResNet18_DSD

ResNet50_DSD

Single-crop (224x224) validation error rate:

The beseline of AlexNet, VGG16, GoogleNet, SqueezeNet are from Caffe Model Zoo. The baseline of ResNet18, ResNet50 are from fb.resnet.torch commit 500b698.