CVPR, 2024
Xiangtai Li · Haobo Yuan . Wei Li . Henghui Ding · Size Wu · Wenwei Zhang ·
Yining Li . Kai Chen . Chen Change Loy*

S-Lab, MMlab@NTU, Shanghai AI Laboratory

In this work, we address various segmentation tasks, each traditionally tackled by distinct or partially unified models. We propose OMG-Seg, One Model that is Good enough to efficiently and effectively handle all the segmentation tasks, including image semantic, instance, and panoptic segmentation, as well as their video counterparts, open vocabulary settings, prompt-driven, interactive segmentation like SAM, and video object segmentation. To our knowledge, this is the first model to fill all these tasks in one model and achieve good enough performance.

We show that OMG-Seg, a transformer-based encoder-decoder architecture with task-specific queries and outputs, can support over ten distinct segmentation tasks and yet significantly reduce computational and parameter overhead across various tasks and datasets. We rigorously evaluate the inter-task influences and correlations during co-training. Both the code and models will be publicly available.

• Test Models and Code are released!!

• Release training code. (to do)
• Release CKPTs.（done）
• Support HuggingFace. (done)

• The first universal model that support image segmentation, video segmentation, open-vocabulary segmentation, multi-dataset segmentation, interactive segmentation.
• A new unified view for solving multiple segmentation tasks in one view.

See DATASET.md

Our codebase is built with MMdetection-3.0 tools.

See INSTALL.md

1. First set up the dataset and environment.

2. Download pre-trained CLIP backbone. The scripts will

3. Generate CLIP text embedding for each dataset. See the embedding generation.

To be released and please stay tuned.

See the configs under seg/configs/m2ov_val.

Test Cityscape dataset, we observe 0.3% noises for Cityscapes panoptic segmentation

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_300q_ov_cityscapes.py 4 --checkpoint model_path

Test COCO dataset, we observe 0.5% noises for COCO panoptic segmentation

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_300q_ov_coco.py 4 --checkpoint model_path

Test Open-Vocabulary ADE dataset, we observe 0.8% noises for COCO panoptic segmentation

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_300q_ov_ade.py 4 --checkpoint model_path

Test Interactive COCO segmentation:

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_ov_coco_pan_point.py 4 --checkpoint model_path

Test Youtube-VIS-19 dataset

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_300q_ov_y19.py 4 --checkpoint model_path

Test VIP-Seg dataset

./tools/dist.sh test seg/configs/m2ov_val/eval_m2_convl_300q_ov_vipseg.py 1 --checkpoint model_path

ConvNeXt-large backbone. model

ConvNeXt-XX-large backbone. model

We will release more models in future.

If you think OMG-Seg codebase are useful for your research, please consider referring us:

@article{li2024omg,
  title={OMG-Seg: Is One Model Good Enough For All Segmentation?},
  author={Li, Xiangtai and Yuan, Haobo and Li, Wei and Ding, Henghui and Wu, Size and Zhang, Wenwei and Li, Yining and Chen, Kai and Loy, Chen Change},
  journal={CVPR},
  year={2024}
}

S-Lab LICENSE.