A Survey on Deep Learning Technique for Video Segmentation

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023
Tianfei Zhou , Fatih Porikli , David Crandall , Luc Van Gool , Wenguan Wang

This repo compiles a collection of resources on deep video segmentation, and will be continuously updated to track developments in the field. Please feel free to submit a pull request if you find any work missing.

1. Introduction

Video segmentation, i.e., partitioning video frames into multiple segments or objects, plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to virtual background creation in video conferencing. In this survey, we comprehensively review two basic lines of research — video object segmentation and video semantic segmentation — by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. In particular, we review eight sub-fields as given in the following figure:

2. Deep Learning-based Video Object Segmentation

• 2.1 Automatic Video Object Segmentation (AVOS)
  • 2.1.1 Deep Learning Module based
  • 2.1.2 Pixel Instance Embedding based
  • 2.1.3 Short-term Information Encoding
  • 2.1.4 Long-term Context Encoding
  • 2.1.5 Un-/Weakly-supervised based
  • 2.1.6 Instance-level AVOS
• 2.2 Semi-automatic Video Object Segmentation (SVOS)
  • 2.2.1 Online Fine-tuning based
  • 2.2.2 Propagation based
  • 2.2.3 Matching based
  • 2.2.4 Box-initialization based
  • 2.2.5 Un-/Weakly-supervised based
  • 2.2.6 Other Specific Methods
• 2.3 Interactive Video Object Segmentation (IVOS)
  • 2.3.1 Interaction-propagation based
  • 2.3.2 Other Methods
• 2.4 Language-guided Video Object Segmentation (LVOS)
  • 2.4.1 Dynamic Convolution based
  • 2.4.2 Capsule Routing based
  • 2.4.3 Attention based

3. Deep Learning-based Video Semantic Segmentation

• 3.1 (Instance-agnostic) Video Semantic Segmentation (VSS)
  • 3.1.1 Efforts towards More Accurate Segmentation
  • 3.1.2 Efforts towards Faster Segmentation
  • 3.1.3 Semi-/Weakly-supervised based
• 3.2 Video Instance Segmentation (VIS)
• 3.3 Video Panoptic Segmentation (VPS)

4. Datasets

• Popular Datasets in VOS and VSS

Citation

If you find our survey and repository useful for your research, please consider citing our paper:

@article{zhou2023survey,
  title={A Survey on Deep Learning Technique for Video Segmentation},
  author={Zhou, Tianfei and Porikli, Fatih and Crandall, David J and Van Gool, Luc and Wang, Wenguan},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2023},
  publisher={IEEE}
}