SFSegNets(ECCV-2020-oral) and SFNet-Lite (Extension)
Reproduced Implementation of Our ECCV-2020 oral paper: Semantic Flow for Fast and Accurate Scene Parsing.
Extension: SFNet-Lite achieve 78.8 mIoU while running 120 FPS, 80.1 mIoU while running at 50 FPS on TITAN-RTX.
Extension: SFNet-Lite achieve new state-of-the-art results (best speed and accuracy trade-off) on domain agnostic driving
segmentation benchmark (Unified Driving Segmentation).
SFNet is the first real time nework which achieves the 80 mIoU on Cityscape test set!!!! It also contains our another concurrent work: SRNet-IEEE-TIP:link.
Our methods achieve the best speed and accuracy trade-off on multiple scene parsing datasets.
Note that the original paper link is on TorchCV where you can train SFnet models.
However, that repo is over-complex for further research and exploration.
Question and Dissussion
If you have any question and or dissussion on fast segmentation, just open an issue. I will reply asap if I have the spare time.
DataSet Setting
Please see the DATASETs.md for the details.
Requirements
pytorch >=1.4.0 apex opencv-python mmcv-cpu
Pretrained models and Trained CKPTs
Please download the pretrained models and put them into the pretrained_models dir on the root of this repo.
pretrained imagenet models
resnet101-deep-stem-pytorch:link
resnet50-deep-stem-pytorch:link
resnet18-deep-stem-pytorch:link
dfnetv1:link
dfnetv2:link
stdcv1/stdc2:link
trained ckpts:
SFNet ckpts:
Cityscape:
sf-resnet18-Mapillary:link
Please download the trained model, the mIoU is on Cityscape validation dataset.
resnet18(no-balanced-sample): 78.4 mIoU
resnet18: 79.0 mIoU link +dsn link
resnet18 + map: 79.9 mIoU link
resnet50: 80.4 mIoU link
dfnetv1: 72.2 mIoU link
dfnetv2: 75.8 mIoU link
SFNet-Lite ckpts:
Cityscape:
sfnet_lite_r18: link
sfnet_lite_r18_coarse_boost: link
sfnet_lite_stdcv2: link
sfnet_lite_stdcv1: link
Unified Driving Segmentation dataset ckpts:
sfnet_lite_r18: link
sfnet_lite_stdcv1: link
sfnet_lite_stdcv2: link
IDD dataset
sfnet_lite_r18: link
sfnet_lite_stdcv1: link
sfnet_lite_stdcv2: link
BDD dataset
sfnet_lite_r18: link
sfnet_r18: link
Mapillary dataset
to be release.
Demo
Visualization Results
python demo_folder.py --arch choosed_architecture --snapshot ckpt_path --demo_floder images_folder --save_dir save_dir_to_disk
Training
All the models are trained with 8 GPUs. The train settings require 8 GPU with at least 11GB memory. Please download the pretrained models before training.
Train ResNet18 model on Cityscapes
SFNet r18
sh ./scripts/cityscapes/train_cityscapes_sfnet_res18.shSFNet-Lite r18
sh ./scripts/cityscapes/train_cityscapes_sfnet_res18_v2_lite_1000e.shTrain ResNet101 models
sh ./scripts/cityscapes/train_cityscapes_sfnet_res101.shSubmission for test
sh ./scripts/submit_test_cityscapes/submit_cityscapes_sfnet_res101.shTrain the Domain Agnostic SFNet for UDS dataset.
Please use the DATASETs.md to prepare the UDS dataset.
sh ./scripts/uds/train_merged_sfnet_res18_v2.shCitation
If you find this repo is useful for your research, Please consider citing our paper:
@article{Li2022SFNetFA,
title={SFNet: Faster, Accurate, and Domain Agnostic Semantic Segmentation via Semantic Flow},
author={Xiangtai Li and Jiangning Zhang and Yibo Yang and Guangliang Cheng and Kuiyuan Yang and Yu Tong and Dacheng Tao},
journal={ArXiv},
year={2022},
volume={abs/2207.04415}
}
@inproceedings{sfnet,
title={Semantic Flow for Fast and Accurate Scene Parsing},
author={Li, Xiangtai and You, Ansheng and Zhu, Zhen and Zhao, Houlong and Yang, Maoke and Yang, Kuiyuan and Tong, Yunhai},
booktitle={ECCV},
year={2020}
}
@article{Li2020SRNet,
title={Towards Efficient Scene Understanding via Squeeze Reasoning},
author={Xiangtai Li and Xia Li and Ansheng You and Li Zhang and Guang-Liang Cheng and Kuiyuan Yang and Y. Tong and Zhouchen Lin},
journal={IEEE-TIP},
year={2021},
}
Acknowledgement
This repo is based on Semantic Segmentation from NVIDIA and DecoupleSegNets
Great Thanks to SenseTime Research for Reproducing All these model ckpts and pretrained model.
License
MIT