NEW! TFDS API for Meta-Dataset

To accompany the presentation of the VTAB+MD paper at NeurIPS 2021's Datasets and Benchmarks track, we are releasing a TensorFlow Datasets-based implementation of Meta-Dataset's input pipeline which is compatible with both the original Meta-Dataset protocol (MD-v1) and the updated protocol designed for VTAB+MD (MD-v2). See the documentation page for more information and example code snippets.

Meta-Dataset

This repository contains accompanying code for the article introducing Meta-Dataset, arxiv.org/abs/1903.03096 and the follow-up paper that proposes the VTAB+MD merged benchmark arxiv.org/abs/2104.02638. It also contains accompanying code and checkpoints for the CrossTransformers https://arxiv.org/abs/2007.11498 and FLUTE https://arxiv.org/abs/2105.07029 follow-up works, which improve performance.

This code is provided here in order to give more details on the implementation of the data-providing pipeline, our back-bones and models, as well as the experimental setting.

See below for user instructions, including how to:

1. install the software,
2. download and convert the data, and
3. train implemented models.

See this introduction notebook for a demonstration of how to sample data from the pipeline (episodes or batches).

In order to run the experiments described in the first version of the arXiv article, arxiv.org/abs/1903.03096v1, please use the instructions, code, and configuration files at version arxiv_v1 of this repository.

We are currently working on updating the instructions, code, and configuration files to reproduce the results in the second version of the article, arxiv.org/abs/1903.03096v2. You can follow the progess in branch arxiv_v2_dev of this repository.

This is not an officially supported Google product.

Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

CrossTransformers: spatially-aware few-shot transfer

Carl Doersch, Ankush Gupta, Andrew Zisserman

This is a Transformer-based neural network architecture which can find coarse spatial correspondence between the query and the support images, and then infer class membership by computing distances between spatially-corresponding features. The paper also introduces SimCLR episodes, which are episodes that require SimCLR-style instance recognition, and therefore encourage features which capture more than just the training-set categories. This algorithm is SOTA on Meta-Dataset (train-on-ILSVRC) as of NeurIPS 2020.

Configuration files for CrossTransformers with and without SimCLR episodes (CTX and CTX+SimCLR Eps from the paper) can be found in learn/gin/default/crosstransformer*. We also have pretrained checkpoints for these two configurations: CTX, and CTX+SimCLR Eps, as well as CTX+SimCLR Eps+BOHB Aug. Note that these were retrained from the versions reported in the paper, but their performance should be on-par. The network structure is the same for all three models, and so they can be loaded using either of the CrossTransformer config files.

Learning a Universal Template for Few-shot Dataset Generalization (FLUTE)

_Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin

Few-shot Learning with a Universal TEmplate (FLUTE) is a model designed for the strong generalization challenge of few-shot learning classes from unseen datasets. At the time of publication (ICML 2021), it achieved SOTA on Meta-Dataset (train-on-all). It works by leveraging the training datasets to learn a 'universal template' that can be repurposed to solve diverse test tasks, by appropriately 'filling in the blanks' of the template each time, with an appropriate set of FiLM parameters that are learned with gradient descent in each test task.

Configuration files for training FLUTE, as well as the dataset classifier used in FLUTE's Blender network can be found in learn/gin/default/flute.gin and learn/gin/default/flute_dataset_classifier.gin, respectively. Configuration files for testing different variants of FLUTE can be found in learn/gin/best/flute* The results reported in the paper were obtained with learn/gin/best/flute.gin.

The training script for FLUTE is train_flute.py. We also have pre-trained checkpoints for FLUTE and its Blender network: https://console.cloud.google.com/storage/gresearch/flute

Leaderboard (in progress)

The tables below were generated by this notebook.

Adding a new model to the leaderboard

1. Gather accuracy results and 95% confidence intervals, as well as the number of episodes used for the CI (minimum 600).
2. If you were affected by #54, make sure the evaluation on Traffic Sign is done on shuffled samples. We encourage you to re-train your best model (or at least perform validation again) as well.
3. Create an issue, with the name of the model, results, as well as the article to cite or any other relevant information to include, and label it "leaderboard". Alternatively, submit a PR with an update to the notebook.

Training on ImageNet only

Training on all datasets

References

[1] Triantafillou et al. (2020)

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle; Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples; ICLR 2020.

[2] Requeima et al. (2019)

James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, Richard E. Turner; Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes; NeurIPS 2019.

[3] Baik et al. (2020)

Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, Kyoung Mu Lee; Meta-Learning with Adaptive Hyperparameters; NeurIPS 2020.

[4] Doersch et al. (2020)

Carl Doersch, Ankush Gupta, Andrew Zisserman; CrossTransformers: spatially-aware few-shot transfer; NeurIPS 2020.

[5] Saikia et al. (2020)

Tonmoy Saikia, Thomas Brox, Cordelia Schmid; Optimized Generic Feature Learning for Few-shot Classification across Domains; arXiv 2020.

[6] Dvornik et al. (2020)

Nikita Dvornik, Cordelia Schmid, Julien Mairal; Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification; ECCV 2020.

[7] Bateni et al. (2020)

Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, Leonid Sigal; Improved Few-Shot Visual Classification; CVPR 2020.

[8] Bateni et al. (2022a)

Peyman Bateni, Jarred Barber, Jan-Willem van de Meent, Frank Wood; Enhancing Few-Shot Image Classification with Unlabelled Examples; WACV 2022.

[9] Liu et al. (2021a)

Lu Liu, William Hamilton, Guodong Long, Jing Jiang, Hugo Larochelle; Universal Representation Transformer Layer for Few-Shot Image Classification; ICLR 2021.

[10] Triantafillou et al. (2021)

Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin; Learning a Universal Template for Few-shot Dataset Generalization; ICML 2021.

[11] Li et al. (2021a)

Wei-Hong Li, Xialei Liu, Hakan Bilen; Universal Representation Learning from Multiple Domains for Few-shot Classification; ICCV 2021.

[12] Li et al. (2021b)

Wei-Hong Li, Xialei Liu, Hakan Bilen; Cross-domain Few-shot Learning with Task-specific Adapters; arXiv 2021.

[13] Liu et al. (2021b)

Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, Yi Yang; A Multi-Mode Modulator for Multi-Domain Few-Shot Classification; ICCV 2021.

[14] Bateni et al. (2022b)

Bateni Peyman, Jarred Barber, Raghav Goyal, Vaden Masrani, Jan-Willem van de Meent, Leonid Sigal, and Frank Wood.; Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning.; arXiv 2022.

User instructions

Installation

Meta-Dataset is generally compatible with Python 2 and Python 3, but some parts of the code may require Python 3. The code works with TensorFlow 2, although it makes extensive use of tf.compat.v1 internally.

• We recommend you follow these instructions to install TensorFlow.
• A list of packages to install is available in requirements.txt, you can install them using pip.
• Clone the meta-dataset repository. Most command lines start with python -m meta_dataset.<something>, and should be typed from within that clone (where a meta_dataset Python module should be visible).
• To reproduce the CrossTransformers training, you will need data augmentation code from simclr, which is autimatically downloaded by setup.py.

Downloading and converting datasets

Meta-Dataset uses several established datasets, that are available from different sources. You can find below a summary of these datasets, as well as instructions to download them and convert them into a common format.

In addition to the datasets below, the FLUTE paper reported results on 3 extra datasets, following recent work. You can find instructions for downloading and converting those 3 additional datasets (MNIST, CIFAR-10 and CIFAR-100) in the CNAPs repo.

For brevity of the command line examples, we assume the following environment variables are defined:

• $DATASRC: root of where the original data is downloaded and potentially extracted from compressed files. This directory does not need to be available after the data conversion is done.
• $SPLITS: directory where *_splits.json files will be created, one per dataset. For instance, $SPLITS/fungi_splits.json contains information about which classes are part of the meta-training, meta-validation, and meta-test set. These files are only used during the dataset conversion phase, but can help troubleshooting later. To re-use the canonical splits instead of re-generating them, you can make it point to meta_dataset/dataset_conversion in your checkout.
• $RECORDS: root directory that will contain the converted datasets (one per sub-directory). This directory needs to be available during training and evaluation.

Dataset summary

Meta-Dataset-v2

In order to make the combined benchmark (VTAB+MD) compatible with each other, Meta-Dataset-v2 makes some changes on the existing pipelines. When converting the ImageNet dataset please use ilsvrc\_2012\_v2 (instructions) in order to make it a training only dataset. Also,VGG Flowers is reserved as a VTAB task in VTAB+MD, so there is no need to convert it. For more details check the paper.

In order to run existing meta-learners with the updated training, validation and test classes you can refer to the learn/gin/setups/imagenet_v2.gin learn/gin/setups/all_v2.gin. These files are meant to be drop in replacements for learn/gin/setups/imagenet.gin and learn/gin/setups/all.gin files respectively.

Training

Experiments are defined via gin configuration files, that are under meta_dataset/learn/gin/:

• setups/ contain generic setups for classes of experiment, for instance which datasets to use (imagenet or all), parameters for sampling the number of ways and shots of episodes.
• models/ define settings for different meta-learning algorithms (baselines, prototypical networks, MAML...)
• default/ contains files that each correspond to one experiment, mostly defining a setup and a model, with default values for training hyperparameters.
• best/ contains files with values for training hyperparameters that achieved the best performance during hyperparameter search.

There are three main architectures, also called "backbones" (or "embedding networks"): four_layer_convnet (sometimes convnet for short), resnet, and wide_resnet. These architectures can be used by all baselines and episodic models. Another backbone, relationnet_convnet (similar to four_layer_convnet but without pooling on the last layer), is only used by RelationNet (and baseline, for pre-training purposes). CrossTransformers use a larger backbone resnet34, which is similar to resnet but with more layers.

Reproducing results

See Reproducing best results for instructions to launch training experiments with the values of hyperparameters that were selected in the paper. The hyperparameters (including the backbone, whether to train from scratch or from pre-trained weights, and the number of training updates) were selected using only the validation classes of the ILSVRC 2012 dataset for all experiments. Even when training on "all" datasets, the validation classes of the other datasets were not used.

Adding task_adaptation code to the path

In order to use data.read_episodes module you need to get task_adaptation code. You can do that by running following code.

git clone https://github.com/google-research/task_adaptation.git
export PYTHONPATH=$PYTHONPATH:$PWD