Apollo

Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

This is the Pytorch implementation for Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

Table of Contents

• Requirements
• Installation
• Notes
• Experiments
• Discussion

Requirements

• Python >= 3.6
• Pytorch >= 1.5.0
• apex
• lmdb >= 0.94
• overrides
• tqdm

Installation

1. Install NVIDIA-apex.
2. Install Pytorch and torchvision

Notes

• In the latest version of Apollo, we changed sigma from 1.0 to 0.01 to make its learning rate in a suitable range, not that different with previous algorithms (see out paper for details). To apply Apollo to your tasks, a reasonable set of hyper parameters to begin with is lr=0.01, eps=1e-4, init_lr=1e-5, warmup=500.
• Warmup plays a super important role for Apollo. Please set warmup to at least 100 updates to achieve stable convergence.

Experimental Results

Image Classification

We use ResNet-110 for CIFAR-10 and standard ResNext-50 for ImageNet. Note that ResNet-110 is a modified version of ResNet-18 to adapt the small image size 32x32 in CIFAR-10. ResNet-110 is much smaller than ResNet-18, with 1.73M parameters (ResNet-18 has 11.69M parameters).

The following table summarizes the key hyper-parameters for different optimizers. For the model training of image classification, please go to this folder.

ResNet-110 on CIFAR-10

ResNext-50 on ImageNet

Note that decoupled weight decay is applied to Adam, RAdam and AdaBelief.

Language Modeling

We use 2-layer LSTMs with 2048 hidden size on One Billion Words. Some key hyper-parameters are listed in the following table. For the model training of language modeling, please go to this folder.

2-layer LSTM on One Billion Words

Since the weight decay rate is zero for all the optimizers, there is no difference between standard L2 regularization and decoupled weight decay.

Neural Machine Translation

We use the Transformer-base models. Some key hyper-parameters are listed in the following table. For the details of NMT experiments, please go to this repo.

Transformer-base on WMT-14 En-De

Discussion

1. Weight Decay:

• The strength of weight decay has significant impact on both the performance of convergence speed and generalization accuracy. Thus, as discussed in the paper, we suggest to consider the effect of regularization strength when we analyze the performance of different optimization methods.

• For adaptive optimizers, including Adam, RAdam and AdaBelief, different implementations of weight decay, such as the decoupled version, lead to very different regularization strength with the same weight decay rate.

• In this paper, for fair comparison, we comprehensively tune the learning rate and the weight decay rate for all the optimizers on CIFAR-10. For ImageNet, due to the resource limits, we kept all the hyper-parameters selected from CIFAR-10 for each optimizer, and only tuned the weight decay rate. One motivation of this is to test the consistency of hyper-parameters of these optimizers on different tasks.

• We analyzed the effect of different weight decay rates on different optimizers. As illustrated in the figure, Apollo achieves improvements over all the three baselines on convergence speed with different rates of weight decay.

2. Epsilon:

We found that AdaBelief is very sensitive to the value of epsilon. In our experiments, Adam, RAdam and Apollo used a fixed epsilon for different tasks (1e-8 for Adam and RAdam, and 1e-4 for Apollo). But for AdaBelief, we had to finetune epsilon for different tasks, e.g. 1e-8 for image classification, 1e-12 for language modeling, and 1e-16 for neural machine translation. With other values, the results of AdaBelief are even worse than Adam (e.g, we tried 1e-8 and 1e-16 for language modeling and the PPL points are higher than 37). Thus, we suspected that the improvements of AdaBelief over Adam or RAdam mainly come from the fine-tuning of epsilon. Similar observations were reported in the EAdam paper.