Multimodal Model-Agnostic Meta-Learning for Few-shot Classification

This project is an implementation of Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation, which is published in NeurIPS 2019. Please visit our project page for more information and contact Shao-Hua Sun for any questions.

Model-agnostic meta-learners aim to acquire meta-prior parameters from a distribution of tasks and adapt to novel tasks with few gradient updates. Yet, seeking a common initialization shared across the entire task distribution substantially limits the diversity of the task distributions that they are able to learn from. We propose a multimodal MAML (MMAML) framework, which is able to modulate its meta-learned prior according to the identified mode, allowing more efficient fast adaptation. An illustration of the proposed framework is as follows.

We evaluate our model and baselines (MAML and Multi-MAML) on multiple multimodal settings based on the following five datasets: (a) Omniglot, (b) Mini-ImageNet, (c) FC100 (e.g. CIFAR100), (d) CUB-200-2011, and (e) FGVC-Aircraft.

Datasets

Run the following command to download and preprocess the datasets

python download.py --dataset aircraft bird cifar miniimagenet

Getting started

Please first install the following prerequisites: wget, unzip.

To avoid any conflict with your existing Python setup, and to keep this project self-contained, it is suggested to work in a virtual environment with virtualenv. To install virtualenv:

pip install --upgrade virtualenv

Create a virtual environment, activate it and install the requirements in requirements.txt.

virtualenv mmaml_venv
source mmaml_venv/bin/activate
pip install -r requirements.txt

Usage

After downloading the datasets, we can start to train models with the following commands.

Training command

$ python main.py -dataset multimodal_few_shot --multimodal_few_shot omniglot miniimagenet cifar bird aircraft --mmaml-model True --num-batches 600000 --output-folder mmaml_5mode_5w1s

• Selected arguments (see the trainer.py for more details)
  • --output-folder: a nickname for the training
  • --dataset: choose among omniglot, miniimagenet, cifar, bird (CUB), and aircraft. You can also add your own datasets.
  • Checkpoints: specify the path to a pre-trained checkpoint
    • --checkpoint: load all the parameters (e.g. train_dir/mmaml_5mode_5w1s/maml_gatedconv_60000.pt).
  • Hyperparameters
    • --num-batches: number of batches
    • --meta-batch-size: number of tasks per batch
    • --slow-lr: learning rate for the global update of MAML
    • --fast-lr: learning rate for the adapted models
    • --num-updates: how many update steps in the inner loop
    • --num-classes-per-batch: how many classes per task (N-way)
    • --num-samples-per-class: how many samples per class for training (K-shot)
    • --num-val-samples: how many samples per class for validation
    • --max_steps: the max training iterations
  • Logging
    • --log-interval: number of batches between tensorboard writes
    • --save-interval: number of batches between model saves
  • Model
    • maml-model: set to True to train a MAML model
    • mmaml-model: set to True to train a MMAML (our) model

Interpret TensorBoard

Launch Tensorboard and go to the specified port, you can see differernt accuracies and losses in the scalars tab.

You can reproduce our results with the following training commands.

2 Modes (Omniglot and Mini-ImageNet)

3 Modes (Omniglot, Mini-ImageNet, and FC100)

5 Modes (Omniglot, Mini-ImageNet, FC100, Aircraft, and CUB)

Multi-MAML

Results

2 Modes (Omniglot and Mini-ImageNet)

3 Modes (Omniglot, Mini-ImageNet, and FC100)

5 Modes (Omniglot, Mini-ImageNet, FC100, Aircraft, and CUB)

Please check out our paper for more comprehensive results.

Related work

• [MAML] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks in ICML 2017
• Probabilistic Model-Agnostic Meta-Learning in NeurIPS 2018
• Bayesian Model-Agnostic Meta-Learning in NeurIPS 2018
• Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace in ICML 2018
• Reptile: A Scalable Meta-Learning Algorithm
• Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples In Meta-Learning Workshop at NeurIPS 2018
• TADAM: Task dependent adaptive metric for improved few-shot learning in NeurIPS 2018
• ProMP: Proximal Meta-Policy Search in ICLR 2019

Cite the paper

If you find this useful, please cite

@inproceedings{vuorio2019multimodal,
  title={Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation},
  author={Vuorio, Risto and Sun, Shao-Hua and Hu, Hexiang and Lim, Joseph J.},
  booktitle={Neural Information Processing Systems},
  year={2019},
}

Authors

Shao-Hua Sun, Risto Vuorio, Hexiang Hu