DETReg: Unsupervised Pretraining with Region Priors for Object Detection (CVPR 2022)

Amir Bar, Xin Wang, Vadim Kantorov, Colorado J Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, Amir Globerson

This repository is the implementation of DETReg, see Project Page.

Introduction

Recent self-supervised pretraining methods for object detection largely focus on pretraining the backbone of the object detector, neglecting key parts of detection architecture. Instead, we introduce DETReg, a new self-supervised method that pretrains the entire object detection network, including the object localization and embedding components. During pretraining, DETReg predicts object localizations to match the localizations from an unsupervised region proposal generator and simultaneously aligns the corresponding feature embeddings with embeddings from a self-supervised image encoder. We implement DETReg using the DETR family of detectors and show that it improves over competitive baselines when finetuned on COCO, PASCAL VOC, and Airbus Ship benchmarks. In low-data regimes, including semi-supervised and few-shot learning settings, DETReg establishes many state-of-the-art results, e.g., on COCO we see a +6.0 AP improvement for 10-shot detection and +3.5 AP improvement when training with only 1% of the labels.

Demo

Interact with the DETReg pretrained model in a Google Colab!

Installation

Requirements

• Linux, CUDA>=9.2, GCC>=5.4

• Python>=3.7

  We recommend you to use Anaconda to create a conda environment:

  conda create -n detreg python=3.7 pip

  Then, activate the environment:

  conda activate detreg

  Installation: (change cudatoolkit to your cuda version. For detailed pytorch installation instructions click here)

  conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch

• Other requirements

  pip install -r requirements.txt

Compiling CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

Usage

Dataset preparation

ImageNet/ImageNet100

Download ImageNet and organize it in the following structure:

code_root/
└── data/
    └── ilsvrc/
          ├── train/
          └── val/

Note that in this work we also used the ImageNet100 dataset, which is x10 smaller than ImageNet. To create ImageNet100 run the following command:

mkdir -p data/ilsvrc100/train
mkdir -p data/ilsvrc100/val
code_root=/path/to/code_root
while read line; do ln -s "${code_root}/data/ilsvrc/train/$line" ${code_root}/data/ilsvrc100/train/$line"; done < "${code_root}/datasets/category.txt"
while read line; do ln -s "${code_root}/data/ilsvrc/val/$line" "${code_root}/data/ilsvrc100/val/$line"; done < "${code_root>/datasets/category.txt"

This should results with the following structure:

code_root/
└── data/
    ├── ilsvrc/
          ├── train/
          └── val/
    └── ilsvrc100/
          ├── train/
          └── val/

MSCoco

Please download COCO 2017 dataset and organize it in the following structure:

code_root/
└── data/
    └── MSCoco/
        ├── train2017/
        ├── val2017/
        └── annotations/
        	├── instances_train2017.json
        	└── instances_val2017.json

Pascal VOC

Download Pascal VOC dataset (2012trainval, 2007trainval, and 2007test):

mkdir -p data/pascal
cd data/pascal
wget http://host.robots.ox.ac.uk:8080/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
tar -xvf VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_06-Nov-2007.tar
tar -xvf VOCtest_06-Nov-2007.tar

The files should be organized in the following structure:

code_root/
└── data/
    └── pascal/
        └── VOCdevkit/
        	├── VOC2007
        	└── VOC2012

Pretraining on ImageNet

The command for pretraining DETReg, based on Deformable-DETR, on 8 GPUs on ImageNet is as follows:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in.sh --batch_size 24 --num_workers 8

Using underlying DETR architecture:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in_detr.sh --batch_size 24 --num_workers 8

The command for pretraining DETReg on 8 GPUs on ImageNet100 is as following:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_in100.sh --batch_size 24 --num_workers 8

Training takes around 1.5 days with 8 NVIDIA V100 GPUs, you can download a pretrained model (see below) if you want to skip this step.

After pretraining, a checkpoint is saved in exps/DETReg_top30_in/checkpoint.pth. To fine tune it over different coco settings use the following commands:

Pretraining on MSCoco

The command for pretraining DETReg on 8 GPUs on MSCoco is as following:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_top30_coco.sh --batch_size 24 --num_workers 8

Finetuning on MSCoco from ImageNet pretraining

Fine tuning on full COCO (should take 2 days with 8 NVIDIA V100 GPUs):

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_full_coco.sh

This assumes a checkpoint exists in exps/DETReg_top30_in/checkpoint.pth.

Finetuning on MSCoco low-data regime, from full MSCoco pretraining (Semi-Supervised Learning setting)

Fine tuning on 1%

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_1pct_coco.sh --batch_size 3

Fine tuning on 2%

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_2pct_coco.sh --batch_size 3

Fine tuning on 5%

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_5pct_coco.sh --batch_size 3

Fine tuning on 10%

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_10pct_coco.sh --batch_size 3

Finetuning on Pascal VOC

Fine tune on full Pascal:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh 8 ./configs/DETReg_fine_tune_full_pascal.sh --batch_size 4 --epochs 100 --lr_drop 70

Fine tune on 10% of Pascal:

GPUS_PER_NODE=2 ./tools/run_dist_launch.sh 2 ./configs/DETReg_fine_tune_10pct_pascal.sh --batch_size 4 --epochs 200 --lr_drop 150

Few-Shot object detection

For few-shot, please follow this code base for the dataloaders, classes, datasplits, etc. We used the few-shot dataset generated with seed = 0.

Using base classes (Table 3)

Finetune DETReg on base classes (60 classes, 99k labeled images). Similar hyperparams as in MSCoco finetuning. Then fine-tune it on few-shot labeled images (80 classes, every class has 10 or 30 instances) for 150 epochs, lr drop after 140 epochs.

No base classes (Table 3)

Fine-tune DETReg on few-shot labeled images (80 classes, every class has 10 or 30 instances) for 1000 epochs, lr drop after 990 epochs.

Evaluation

To evaluate a finetuned model, use the following command from the project basedir:

./configs/<config file>.sh --resume exps/<config file>/checkpoint.pth --eval

Pretrained Models Zoo

Citation

If you found this code helpful, feel free to cite our work:

@misc{bar2021detreg,
      title={DETReg: Unsupervised Pretraining with Region Priors for Object Detection},
      author={Amir Bar and Xin Wang and Vadim Kantorov and Colorado J Reed and Roei Herzig and Gal Chechik and Anna Rohrbach and Trevor Darrell and Amir Globerson},
      year={2021},
      eprint={2106.04550},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Related Works

If you found DETReg useful, consider checking out these related works as well: ReSim, SwAV, DETR, UP-DETR, and Deformable DETR.

Change Log

• 07/17/23 - Release DETReg few-shot learning checkpoints pretrained finetuned on baseclass and from scratch (Table 3,4).
• 07/17/23 - Update few-shot learning results. New paper version.
• 04/28/22 - Bug fix in multiprocessing, affects Table 5 results. Up-to-date results here, new paper version will be uploaded tonight.
• 12/13/21 - Add DETR architecture
• 12/12/21 - Update experiments hyperparams in accordance with new paper version
• 12/12/21 - Avoid box caching on TopK policy (bug fix)
• 9/19/21 - Fixed Pascal VOC training with %X of training data

Acknowlegments

DETReg builds on previous works code base such as Deformable DETR and UP-DETR. If you found DETReg useful please consider citing these works as well.

License

DETReg is released under the Apache 2.0 license. Please see the LICENSE file for more information.