OpenOOD: Benchmarking Generalized OOD Detection

This repository reproduces representative methods within the Generalized Out-of-Distribution Detection Framework, aiming to make a fair comparison across methods that were initially developed for anomaly detection, novelty detection, open set recognition, and out-of-distribution detection. This codebase is still under construction. Comments, issues, contributions, and collaborations are all welcomed!

Timeline of the methods that OpenOOD supports. More methods are included as OpenOOD iterates.

Updates

• 16 June, 2023: 💥💥 We are releasing OpenOOD v1.5, which includes the following exciting updates. A detailed changelog is provided in the Wiki.
  • A new report which provides benchmarking results on ImageNet and for full-spectrum detection.
  • A unified, easy-to-use evaluator that allows evaluation by simply creating an evaluator instance and calling its functions. Check out this colab tutorial!
  • A live leaderboard that tracks the state-of-the-art of this field.
• 14 October, 2022: OpenOOD v1.0 is accepted to NeurIPS 2022. Check the report here.
• 14 June, 2022: We release v0.5.
• 12 April, 2022: Primary release to support Full-Spectrum OOD Detection.

FAQ

• APS_mode means Automatic (hyper)Parameter Searching mode, which enables the model to validate all the hyperparameters in the sweep list based on the validation ID/OOD set. The default value is False. Check here for example.

Get Started

v1.5 (up-to-date)

Installation

OpenOOD now supports installation via pip.

pip install git+https://github.com/Jingkang50/OpenOOD

Data

If you only use our evaluator, the benchmarks for evaluation will be automatically downloaded by the evaluator (again check out this tutorial). If you would like to also use OpenOOD for training, you can get all data with our downloading script. Note that ImageNet-1K training images should be downloaded from its official website.

Pre-trained checkpoints

OpenOOD v1.5 focuses on 4 ID datasets, and we release pre-trained models accordingly.

• CIFAR-10 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
• CIFAR-100 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
• ImageNet-200 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
• ImageNet-1K [Google Drive]: ResNet-50 classifiers including 1) the one from torchvision, 2) the ones that are trained by us with specific methods such as MOS, CIDER, and 3) the official checkpoints of data augmentation methods such as AugMix, PixMix.

Again, these checkpoints can be downloaded with the downloading script here.

v1.0 (no longer maintained)

To setup the environment, we use conda to manage our dependencies.

Our developers use CUDA 10.1 to do experiments.

You can specify the appropriate cudatoolkit version to install on your machine in the environment.yml file, and then run the following to create the conda environment:

conda env create -f environment.yml
conda activate openood
pip install libmr==0.1.9 # if necessary

Datasets and pretrained models are provided here. Please unzip the files if necessary. We also provide an automatic data download script here.

Our codebase accesses the datasets from ./data/ and pretrained models from ./results/checkpoints/ by default.

├── ...
├── data
│   ├── benchmark_imglist
│   ├── images_classic
│   ├── images_medical
│   └── images_largescale
├── openood
├── results
│   ├── checkpoints
│   └── ...
├── scripts
├── main.py
├── ...

The easiest hands-on script is to train LeNet-5 on MNIST and evaluate its OOD or FS-OOD performance with MSP baseline.

sh scripts/basics/mnist/train_mnist.sh
sh scripts/ood/msp/mnist_test_ood_msp.sh

Supported Benchmarks (10)

This part lists all the benchmarks we support. Feel free to include more.

Note that OpenOOD v1.5 emphasizes and focuses on the last 4 benchmarks for OOD detection.

Supported Backbones (6)

This part lists all the backbones we will support in our codebase, including CNN-based and Transformer-based models. Backbones like ResNet-50 and Transformer have ImageNet-1K/22K pretrained models.

Supported Methods (50+)

This part lists all the methods we include in this codebase. Up to v1.5, we totally support more than 50 popular methods for generalized OOD detection.

All the supported methodolgies can be placed in the following four categories.

We also note our supported methodolgies with the following tags if they have special designs in the corresponding steps, compared to the standard classifier training process.

Contributing

We appreciate all contributions to improve OpenOOD. We sincerely welcome community users to participate in these projects. Please refer to CONTRIBUTING.md for the contributing guideline.

Contributors

Citation

If you find our repository useful for your research, please consider citing our paper:

# v1.5 report
@article{zhang2023openood,
  title={OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection},
  author={Zhang, Jingyang and Yang, Jingkang and Wang, Pengyun and Wang, Haoqi and Lin, Yueqian and Zhang, Haoran and Sun, Yiyou and Du, Xuefeng and Zhou, Kaiyang and Zhang, Wayne and Li, Yixuan and Liu, Ziwei and Chen, Yiran and Li, Hai},
  journal={arXiv preprint arXiv:2306.09301},
  year={2023}
}

# v1.0 report
@article{yang2022openood,
    author = {Yang, Jingkang and Wang, Pengyun and Zou, Dejian and Zhou, Zitang and Ding, Kunyuan and Peng, Wenxuan and Wang, Haoqi and Chen, Guangyao and Li, Bo and Sun, Yiyou and Du, Xuefeng and Zhou, Kaiyang and Zhang, Wayne and Hendrycks, Dan and Li, Yixuan and Liu, Ziwei},
    title = {OpenOOD: Benchmarking Generalized Out-of-Distribution Detection},
    year = {2022}
}

@article{yang2022fsood,
    title = {Full-Spectrum Out-of-Distribution Detection},
    author = {Yang, Jingkang and Zhou, Kaiyang and Liu, Ziwei},
    journal={arXiv preprint arXiv:2204.05306},
    year = {2022}
}

@article{yang2021oodsurvey,
    title={Generalized Out-of-Distribution Detection: A Survey},
    author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
    journal={arXiv preprint arXiv:2110.11334},
    year={2021}
}