Generalized Out-of-Distribution Detection: A Survey

1. Overview

This repository is with our survey paper:

Title: Generalized Out-of-Distribution Detection: A Survey
Authors: Jingkang Yang¹, Kaiyang Zhou¹, Yixuan Li², Ziwei Liu¹
Institutions: ¹MMLab@NTU, ²University of Wisconsin-Madison.

This survey comprehensively reviews the similar topics of outlier detection (OD), anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and out-of-distribution (OOD) detection, extensively compares their commomality and differences, and eventually unifies them under a big umbrella of "generalized OOD detection" framework.

We hope that this survey can help readers and participants better understand the open-world field centered on OOD detection. At the same time, it urges future work to learn, compare, and develop ideas and methods from the broader scope of generalized OOD detection, with clear problem definition and proper benchmarking.

We prepare this repository for the following two reasons:

1. We consider it an awesome list to easily access the references mentioned in the paper Table 1. We also believe this list will continue to include more promising works as new works appear. Please feel free to nominate good related works with Pull Requests.
2. We hope this repository becomes a discussion panel for readers to ask questions, raise concerns, and make constructive comments for the broad generalized OOD detection field. Please feel free to post your ideas in the Issues.

We are also planning to build an evaluation benchmark to compare representative generalized OOD detection methods from every sub-task to further unify the field. The work will be collaborated with SenseTime EIG Research, which recently have many full-time researcher openings for this benchmarking project and other OOD-related research. Check their Recruitment Info for more information.

Fig.1.1: Two kinds of distribution shift to assist better understanding of our framework.	Fig.1.2: Taxonomy diagram of generalized OOD detection framework.

2. Taxonomy

• 2.1 Anomaly Detection
  • 2.1.1 Sensory Anomaly Detection
  • 2.1.2 Semantic Anomaly Detection
• 2.2 Novelty Detection
  • 2.2.1 One-Class Novelty Detection
  • 2.2.2 Multi-Class Novelty Detection
• 2.3 Open Set Recognition
• 2.4 Out-of-Distribution Detection
• 2.5 Outlier Detection
• 2.6 Discussion

3. Anomaly Detection & One-Class Novelty Detection

• 3.1 Density-based Methods
  • 3.1.1 Classic Density Estimation
  • 3.1.2 NN-based Density Estimation
  • 3.1.3 Energy-based Model
  • 3.1.4 Frequency-based Model
• 3.2 Reconstruction-based Methods
  • 3.2.1 Sparse Representation Methods
  • 3.2.2 Reconstruction-Error Methods
• 3.3 Classification-based Methods
  • 3.3.1 One-Class Classification
  • 3.3.2 Positive-Unlabeled Learning
  • 3.3.3 Self-Supervised Learning
• 3.4 Distance-based Methods
• 3.5 Gradient-based Methods
• 3.6 Discussion and Theoretical Analysis

4. Multi-Class Novelty Detection & Open Set Recognition

• 4.1 Classfication-based Methods
  • 4.1.1 EVT-based Calibration
  • 4.1.2 EVT-free Calibration
  • 4.1.3 Unknown Class Generation
  • 4.1.4 Label Space Redesign
• 4.2 Distance-based Methods
• 4.3 Reconstruction-based Methods
  • 4.3.1 Sparse Representation Methods
  • 4.3.2 Reconstruction-Error Methods

5. Out-of-Distribution Detection

• 5.1 Classfication-based Methods
  • 5.1.0 Baseline
  • 5.1.1 Output-based Methods
    • 5.1.1.1 Post-hoc Detection
    • 5.1.1.2 Confidence Enhancement
    • 5.1.1.3 Outlier Exposure
  • 5.1.2 OOD Data Generation
  • 5.1.3 Gradient-based Methods
  • 5.1.4 Bayesian Models
  • 5.1.5 Large-scale OOD Detection
• 5.2 Density-based Methods
• 5.3 Distance-based Methods

6. Outlier Detection

• 6.1 Density-based Methods
• 6.2 Distance-based Methods
  • 6.2.1 Cluster-based Methods
  • 6.2.2 Graph-based Methods
• 6.3 Classification-based Methods

7. Challenges and Future Direction

• 7.1 Challenges
  • 7.1.1 Proper Evaluation and Benchmarking
  • 7.1.2 Outlier-free OOD Detection
  • 7.1.3 Tradeoff Between Classification and OOD Detection
  • 7.1.4 Real-world Benchmarks and Evaluations
• 7.2 Future Directions
  • 7.2.1 Methodologies across Sub-tasks
  • 7.2.2 OOD Detection & OOD Generalization
  • 7.2.3 OOD Detection & Open-Set Noisy Labels
  • 7.2.4 Theoretical Analysis

8. Conclusion

In this survey, we comprehensively review five topics: AD, ND, OSR, OOD detection, and OD, and unify them as a framework of generalized OOD detection. By articulating the motivations and definitions of each sub-task, we encourage follow-up works to accurately locate their target problems and find the most suitable benchmarks. By sorting out the methodologies for each sub-task, we hope that readers can easily grasp the mainstream methods, identify suitable baselines, and contribute future solutions in light of existing ones. By providing insights, challenges, and future directions, we hope that future works will pay more attention to the existing problems and explore more interactions across other tasks within or even outside the scope of generalized OOD detection.

Citation

If you find our survey and repository useful for your research, please consider citing our paper:

@article{yang2021oodsurvey,
  title={Generalized Out-of-Distribution Detection: A Survey},
  author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
  journal={arXiv preprint arXiv:2110.11334},
  year={2021}
}

Acknowledgements

This repository is created and maintained by Jingkang Yang and Peng Wenxuan from NTU; Kunyuan Ding, Zixu Song, Pengyun Wang, Zitang Zhou, and Dejian Zou from BUPT.