IntelLabs/distiller

Clone the Distiller code repository from github:

$ git clone https://github.com/IntelLabs/distiller.git

The rest of the documentation that follows, assumes that you have cloned your repository to a directory called distiller.

We recommend using a Python virtual environment, but that of course, is up to you. There's nothing special about using Distiller in a virtual environment, but we provide some instructions, for completeness.
Before creating the virtual environment, make sure you are located in directory distiller. After creating the environment, you should see a directory called distiller/env.

Using virtualenv

If you don't have virtualenv installed, you can find the installation instructions here.

To create the environment, execute:

$ python3 -m virtualenv env

This creates a subdirectory named env where the python virtual environment is stored, and configures the current shell to use it as the default python environment.

Using venv

If you prefer to use venv, then begin by installing it:

$ sudo apt-get install python3-venv

Then create the environment:

$ python3 -m venv env

As with virtualenv, this creates a directory called distiller/env.

Activate the environment

The environment activation and deactivation commands for venv and virtualenv are the same.
!NOTE: Make sure to activate the environment, before proceeding with the installation of the dependency packages:

$ source env/bin/activate

Finally, install the Distiller package and its dependencies using pip3:

$ cd distiller
$ pip3 install -e .

This installs Distiller in "development mode", meaning any changes made in the code are reflected in the environment without re-running the install command (so no need to re-install after pulling changes from the Git repository).

Notes:

• Distiller has only been tested on Ubuntu 16.04 LTS, and with Python 3.5.
• If you are not using a GPU, you might need to make small adjustments to the code.

The following will invoke training-only (no compression) of a network named 'simplenet' on the CIFAR10 dataset. This is roughly based on TorchVision's sample Imagenet training application, so it should look familiar if you've used that application. In this example we don't invoke any compression mechanisms: we just train because for fine-tuning after pruning, training is an essential part.

Note that the first time you execute this command, the CIFAR10 code will be downloaded to your machine, which may take a bit of time - please let the download process proceed to completion.

The path to the CIFAR10 dataset is arbitrary, but in our examples we place the datasets in the same directory level as distiller (i.e. ../../../data.cifar10).

First, change to the sample directory, then invoke the application:

$ cd distiller/examples/classifier_compression
$ python3 compress_classifier.py --arch simplenet_cifar ../../../data.cifar10 -p 30 -j=1 --lr=0.01

You can use a TensorBoard backend to view the training progress (in the diagram below we show a couple of training sessions with different LR values). For compression sessions, we've added tracing of activation and parameter sparsity levels, and regularization loss.

We've included in the git repository a few checkpoints of a ResNet20 model that we've trained with 32-bit floats. Let's load the checkpoint of a model that we've trained with channel-wise Group Lasso regularization.
With the following command-line arguments, the sample application loads the model (--resume) and prints statistics about the model weights (--summary=sparsity). This is useful if you want to load a previously pruned model, to examine the weights sparsity statistics, for example. Note that when you resume a stored checkpoint, you still need to tell the application which network architecture the checkpoint uses (-a=resnet20_cifar):

$ python3 compress_classifier.py --resume=../ssl/checkpoints/checkpoint_trained_ch_regularized_dense.pth.tar -a=resnet20_cifar ../../../data.cifar10 --summary=sparsity

You should see a text table detailing the various sparsities of the parameter tensors. The first column is the parameter name, followed by its shape, the number of non-zero elements (NNZ) in the dense model, and in the sparse model. The next set of columns show the column-wise, row-wise, channel-wise, kernel-wise, filter-wise and element-wise sparsities.
Wrapping it up are the standard-deviation, mean, and mean of absolute values of the elements.

In the Compression Insights notebook we use matplotlib to plot a bar chart of this summary, that indeed show non-impressive footprint compression.

Although the memory footprint compression is very low, this model actually saves 26.6% of the MACs compute.

$ python3 compress_classifier.py --resume=../ssl/checkpoints/checkpoint_trained_channel_regularized_resnet20_finetuned.pth.tar -a=resnet20_cifar ../../../data.cifar10 --summary=compute

This example performs 8-bit quantization of ResNet20 for CIFAR10. We've included in the git repository the checkpoint of a ResNet20 model that we've trained with 32-bit floats, so we'll take this model and quantize it:

$ python3 compress_classifier.py -a resnet20_cifar ../../../data.cifar10 --resume ../ssl/checkpoints/checkpoint_trained_dense.pth.tar --quantize-eval --evaluate

The command-line above will save a checkpoint named quantized_checkpoint.pth.tar containing the quantized model parameters. See more examples here.

• DeGirum Pruned Models - a repository containing pruned models and related information.

• TorchFI - TorchFI is a fault injection framework build on top of PyTorch for research purposes.

• hsi-toolbox - Hyperspectral CNN compression and band selection

• Brunno F. Goldstein, Sudarshan Srinivasan, Dipankar Das, Kunal Banerjee, Leandro Santiago, Victor C. Ferreira, Alexandre S. Nery, Sandip Kundu, Felipe M. G. Franca.
  Reliability Evaluation of Compressed Deep Learning Models,
  In IEEE 11th Latin American Symposium on Circuits & Systems (LASCAS), San Jose, Costa Rica, 2020, pp. 1-5.

• Pascal Bacchus, Robert Stewart, Ekaterina Komendantskaya.
  Accuracy, Training Time and Hardware Efficiency Trade-Offs for Quantized Neural Networks on FPGAs,
  In Applied Reconfigurable Computing. Architectures, Tools, and Applications. ARC 2020. Lecture Notes in Computer Science, vol 12083. Springer, Cham

• Indranil Chakraborty, Mustafa Fayez Ali, Dong Eun Kim, Aayush Ankit, Kaushik Roy.
  GENIEx: A Generalized Approach to Emulating Non-Ideality in Memristive Xbars using Neural Networks,
  arXiv:2003.06902, 2020.

• Ahmed T. Elthakeb, Prannoy Pilligundla, Fatemehsadat Mireshghallah, Tarek Elgindi, Charles-Alban Deledalle, Hadi Esmaeilzadeh.
  Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization,
  arXiv:2003.00146, 2020.

• Ziqing Yang, Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, Guoping Hu.
  TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing,
  arXiv:2002.12620, 2020.

• Alexander Kozlov, Ivan Lazarevich, Vasily Shamporov, Nikolay Lyalyushkin, Yury Gorbachev.
  Neural Network Compression Framework for fast model inference,
  arXiv:2002.08679, 2020.

• Moran Shkolnik, Brian Chmiel, Ron Banner, Gil Shomron, Yuri Nahshan, Alex Bronstein, Uri Weiser.
  Robust Quantization: One Model to Rule Them All,
  arXiv:2002.07686, 2020.

• Muhammad Abdullah Hanif, Muhammad Shafique.
  SalvageDNN: salvaging deep neural network accelerators with permanent faults through saliency-driven fault-aware mapping,
  In Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering SciencesVolume 378, Issue 2164, 2019.
  https://doi.org/10.1098/rsta.2019.0164

• Meiqi Wang, Jianqiao Mo, Jun Lin, Zhongfeng Wang, Li Du.
  DynExit: A Dynamic Early-Exit Strategy for Deep Residual Networks,
  In IEEE International Workshop on Signal Processing Systems (SiPS), 2019.

• Vinu Joseph, Saurav Muralidharan, Animesh Garg, Michael Garland, Ganesh Gopalakrishnan.
  A Programmable Approach to Model Compression,
  arXiv:1911.02497, 2019
  code

• Hui Guan, Lin Ning, Zhen Lin, Xipeng Shen, Huiyang Zhou, Seung-Hwan Lim.
  In-Place Zero-Space Memory Protection for CNN,
  In Conference on Neural Information Processing Systems (NeurIPS), 2019.
  arXiv:1910.14479, 2019
  code

• Hossein Baktash, Emanuele Natale, Laurent Viennot.
  A Comparative Study of Neural Network Compression,
  arXiv:1910.11144, 2019.

• Maxim Zemlyanikin, Alexander Smorkalov, Tatiana Khanova, Anna Petrovicheva, Grigory Serebryakov.
  512KiB RAM Is Enough! Live Camera Face Recognition DNN on MCU,
  In IEEE International Conference on Computer Vision (ICCV), 2019.

• Ziheng Wang, Jeremy Wohlwend, Tao Lei.
  Structured Pruning of Large Language Models,
  arXiv:1910.04732, 2019.

• Soroush Ghodrati, Hardik Sharma, Sean Kinzer, Amir Yazdanbakhsh, Kambiz Samadi, Nam Sung Kim, Doug Burger, Hadi Esmaeilzadeh.
  Mixed-Signal Charge-Domain Acceleration of Deep Neural networks through Interleaved Bit-Partitioned Arithmetic,
  arXiv:1906.11915, 2019.

• Gil Shomron, Tal Horowitz, Uri Weiser.
  SMT-SA: Simultaneous Multithreading in Systolic Arrays,
  In IEEE Computer Architecture Letters (CAL), 2019.

• Shangqian Gao , Cheng Deng , and Heng Huang.
  Cross Domain Model Compression by Structurally Weight Sharing,
  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8973-8982.

• Moin Nadeem, Wei Fang, Brian Xu, Mitra Mohtarami, James Glass.
  FAKTA: An Automatic End-to-End Fact Checking System,
  In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.

• Ahmed T. Elthakeb, Prannoy Pilligundla, Hadi Esmaeilzadeh.
  SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training,
  arXiv:1905.01416, 2019. code

• Goncharenko A., Denisov A., Alyamkin S., Terentev E.
  Trainable Thresholds for Neural Network Quantization,
  In: Rojas I., Joya G., Catala A. (eds) Advances in Computational Intelligence Lecture Notes in Computer Science, vol 11507. Springer, Cham. International Work-Conference on Artificial Neural Networks (IWANN 2019).

• Ahmed T. Elthakeb, Prannoy Pilligundla, Hadi Esmaeilzadeh.
  Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks,
  arXiv:1906.06033, 2019

• Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, Zhiru Zhang.
  Improving Neural Network Quantization without Retraining using Outlier Channel Splitting,
  arXiv:1901.09504, 2019
  Code

• Angad S. Rekhi, Brian Zimmer, Nikola Nedovic, Ningxi Liu, Rangharajan Venkatesan, Miaorong Wang, Brucek Khailany, William J. Dally, C. Thomas Gray.
  Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference,
  Nvidia Research, 2019.

• Norio Nakata.
  Recent Technical Development of Artificial Intelligence for Diagnostic Medical Imaging,
  In Japanese Journal of Radiology, February 2019, Volume 37, Issue 2, pp 103–108.

• Alexander Goncharenko, Andrey Denisov, Sergey Alyamkin, Evgeny Terentev.
  Fast Adjustable Threshold For Uniform Neural Network Quantization,
  arXiv:1812.07872, 2018