Training neural networks with local error signals

This repo contains PyTorch code for training neural networks without global backprop. Experiments are performed by Arild Nøkland and Lars Hiller Eidnes.

A more detailed description of the experiments is available on arXiv here: https://arxiv.org/abs/1901.06656

Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network with layer-wise loss functions. In this paper we demonstrate, for the first time, that layer-wise training can approach the state-of-the-art on a variety of image datasets. We use single-layer sub-networks and two different supervised loss functions to generate local error signals for the hidden layers, and we show that the combination of these losses help with optimization in the context of local learning. Using local errors could be a step towards more biologically plausible deep learning because the global error does not have to be transported back to hidden layers.

In the tables below, 'pred' indicates a layer-wise cross-entropy loss, 'sim' indicates a layer-wise similarity matching loss, and 'predsim' indicates a combination of these losses. For the local losses, the computational graph is detached after each hidden layer.

Experiments

Results on MNIST with 2 pixel jittering:

Results on Fashion-MNIST with 2 pixel jittering and horizontal flipping:

Results on Kuzusjiji-MNIST with no data augmentation:

Results on Cifar-10 with data augmentation:

Results on Cifar-100 with data augmentation:

Results on SVHN with extra training data, but no augmentation:

Results on STL-10 with no data augmentation:

Training recipes

To replicate training of MLP on MNIST with local loss 'predsim':

python train.py --model mlp --dataset MNIST --dropout 0.1 --lr 5e-4 --num-layers 3 --epochs 100 --lr-decay-milestones 50 75 89 94 --nonlin leakyrelu

To replicate training of VGG8b on MNIST with local loss 'predsim':

python train.py --model vgg8b --dataset MNIST --dropout 0.2 --lr 5e-4 --epochs 100 --lr-decay-milestones 50 75 89 94 --nonlin leakyrelu --dim-in-decoder 1024

To replicate training of MLP on CIFAR10 with local loss 'predsim':

python train.py --model mlp --dataset CIFAR10 --dropout 0.1 --lr 5e-4 --num-layers 3 --num-hidden 3000 --nonlin leakyrelu

To replicate training of VGG8b on CIFAR10 with local loss 'predsim':

python train.py --model vgg8b --dataset CIFAR10 --dropout 0.2 --lr 5e-4 --nonlin leakyrelu --dim-in-decoder 2048

To replicate training of VGG11b (3x) on CIFAR10 with local loss 'predsim':

python train.py --model vgg11b --dataset CIFAR10 --dropout 0.3 --lr 3e-4 --feat-mult 3 --nonlin leakyrelu

For all the above recipes, to train with local cross-entropy loss, add argument

--loss-sup pred

For all the above recipes, to train with local similarity matching loss, add argument

--loss-sup sim

For all the above recipes, to train with global loss, add argument

--backprop

For all the above recipes, to train with a more biologically plausible version of local loss, add argument

--bio

To add cutout regularization with cutout hole size 14, add arguments

--cutout --length 14

To replicate all the above experiments, run

./run_experiments.sh