Introduction

• We build GV-B(General Vision Benchmark) on Classification, Detection, Segmentation and Depth Estimation including 26 datasets for model evaluation.

• It is recommended to evaluate with low-data regime, using only 10% training data.

• The parameters of model backbone will be frozen during training, as known as 'linear probe'.

• Face Detection and Depth Estimation is not provided for now, you may evaluate via official repo if needed.

• Specifically, we use central_model.py in our repo to represent the implementation of Up-G models.

  🏡 Homepage | 📘 Documentation | 👀 Model Zoo

Task Supported

• Object Classification
• Object Detection (VOC Detection)
• Pedestrian Detection (CityPersons Detection)
• Semantic Segmentation (VOC Segmentation)
• Face Detection (WiderFace Detection)
• Depth Estimation (Kitti/NYU-v2 Depth Estimation)

Installation

Requirements

• Python 3.6+
• PyTorch 1.8+
• openmim==0.1.5
• MMCV>=1.3.16

Install Dependencies

a. Create a conda virtual environment and activate it.

conda create -n open-mmlab python=3.8 -y
conda activate open-mmlab

b. Install PyTorch and torchvision following the official instructions, e.g.：

conda install pytorch torchvision -c pytorch

Make sure that your compilation CUDA version and runtime CUDA version match.
You can check the supported CUDA version for precompiled packages on the
[PyTorch website](https://pytorch.org/).

c. Install openmm package via pip (mmcls, mmdet, mmseg)：

pip install mmcls
pip install mmdet
pip install mmsegmentation

Usage

This section provide basic tutorials about the usage of GV-B.

Prepare datasets

For each evaluation task, you can follow the official repo tutorial for data preparation.

mmclassification (take flowers for example)
├── data
│   ├── flowers
│   │   ├── train/
│   │   ├── test/
│   │   ├── train_meta.list
│   │   ├── test_meta.list

mmdetection
├── data
│   ├── citypersons
│   │   ├── annotations
│   │   ├── leftImg8bit
│   │   │   ├── train
│   │   │   ├── val
│   ├── VOCdevkit
│   │   ├── VOC2007
│   │   ├── VOC2012

mmsegmentation
├── data
│   ├── VOCdevkit
│   │   ├── VOC2012
│   │   │   ├── JPEGImages
│   │   │   ├── SegmentationClass
│   │   │   ├── ImageSets
│   │   │   │   ├── Segmentation
│   │   ├── VOCaug
│   │   │   ├── dataset
│   │   │   │   ├── cls

Model evaluation

We use MIM to submit evaluation in GV-B.

a.If you run on a cluster managed with slurm, you can use the script mim_slurm_train.sh. (This script also supports single machine training.)

sh tools/mim_slurm_train.sh $PARTITION $TASK $CONFIG $WORK_DIR

## mmcls as an example
sh tools/mim_slurm_train.sh GVT mmcls configs/cls/linear_probe/mnb4_Up-E-C_pretrain_flowers_10p.py /path/to/your/project

b.If you run on w/o slurm. (More details can be found in docs of openmim)

PYTHONPATH='.':$PYTHONPATH mim train $TASK $CONFIG $WORK_DIR

• PARTITION: The partition you are using
• WORK_DIR: The directory to save logs and checkpoints
• CONFIG: Config files corresponding to tasks.

Detailed Tutorials

Currently, we provide tutorials (mmclassification for example) for further development.

• add new modules
• add new dataset
• learn about config
• design data pipeline
• customize schedule
• customize runtime settings

Benchmark(with Hyperparameter searching)