• Stars
    star
    699
  • Rank 64,759 (Top 2 %)
  • Language
    Python
  • License
    BSD 2-Clause "Sim...
  • Created over 5 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The official code for the paper 'Structured Knowledge Distillation for Semantic Segmentation'. (CVPR 2019 ORAL) and extension to other tasks.

Structured Knowledge Distillation for Dense Prediction

This repository contains the source code of our paper Structured Knowledge Distillation for Dense Prediction. It is an extension of our paper Structured Knowledge Distillation for Semantic Segmentation (accepted for publication in CVPR'19, oral).

We have update a more stable version of training the GAN part in the master branch.

If you want to transfer our pair-wise distilaltion and pixel-wise distillation in your own work or you want to use our trained models in the conference version, you can checkout to the old branck 'cvpr_19'.

Sample results

Demo video for the student net (ESPNet) on Camvid

After distillation with mIoU 65.1: image

Before distillation with mIoU 57.8: image

Structure of this repository

This repository is organized as:

  • libs This directory contains the inplaceABNSync modes.
  • dataset This directory contains the dataloader for different datasets.
  • network This directory contains a model zoo for network models.
  • utils This directory contains api for calculating the distillation loss.

Performance on the Cityscape dataset

We apply the distillation method to training the PSPNet. We used the dataset splits (train/val/test) provided here. We trained the models at a resolution of 512x512. Pi: Pixel-wise distillation PA: Pair-wise distillation HO: holistic distillation

Model Average
baseline 69.10
+Pi 70.51
+Pi+Pa 71.78
+Pi+Pa+Ho 74.08

Pre-trained model and Performance on other tasks

Pretrain models for three tasks can be found here:

Task Dataset Network Method Evaluation Metric Link
Semantic Segmentation Cityscapes ResNet18 Baseline miou: 69.10 -
Semantic Segmentation Cityscapes ResNet18 + our distillation miou: 75.3 link
Object Detection COCO FCOS-MV2-C128 Baseline mAP: 30.9 -
Object Detection COCO FCOS-MV2-C128 + our distillation mAP: 34.0 link
Depth estimation nyudv2 VNL baseline rel: 13.5 -
Depth estimation nyudv2 VNL + our distillation rel: 13.0 link

Note: Other chcekpoints can be obtained by email: [email protected] if needed.

Requirement

python3.5

pytorch0.4.1

ninja

numpy

cv2

Pillow

We recommend to use Anaconda.

We have tested our code on Ubuntu 16.04.

Compiling

Some parts of InPlace-ABN have a native CUDA implementation, which must be compiled with the following commands:

cd libs
sh build.sh
python build.py

The build.sh script assumes that the nvcc compiler is available in the current system search path. The CUDA kernels are compiled for sm_50, sm_52 and sm_61 by default. To change this (e.g. if you are using a Kepler GPU), please edit the CUDA_GENCODE variable in build.sh.

Quick start to test the model

  1. download the Cityscape dataset
  2. sh run_test.sh [you should change the data-dir to your own]. By using our distilled student model, which can be gotten in [ckpt], an mIoU of 73.05 is achieved on the Cityscape test set, and 75.3 on validation set.
Model Average roda sidewalk building wall fence pole trafficlight trafficsign vegetation terrain sky person rider car truck bus train motorcycle bicycle
IoU 73.05 97.57 78.80 91.42 50.76 50.88 60.77 67.93 73.18 92.49 70.36 94.56 82.81 61.64 94.89 60.14 66.62 59.93 61.50 71.71

Note: Depth estimation task and object detection task can be test through the original projects of VNL and FCOS using our checkpoints.

Train script

Download the pre-trained teacher weight:

If you want to reproduce the ablation study in our paper, please modify is_pi_use/is_pa_use/is_ho_use in the run_train_eval.sh. sh run_train_eval.sh

Test script

If you want to test your method on the cityscape test set, please modify the data-dir and resume-from path to your own, then run the test.sh and submit your results to www.cityscapes-dataset.net/submit/ sh test.sh

License

For academic use, this project is licensed under the 2-clause BSD License - see the LICENSE file for details. For commercial use, please contact Yifan Liu and Chunhua Shen.

More Repositories

1

CoupleGenerator

Generate your lover with your photo
Python
459
star
2

ETC-Real-time-Per-frame-Semantic-video-segmentation

Enforcing temporal consistency in real-time per-frame semantic video segmentation
Python
296
star
3

TorchDistiller

Python
192
star
4

Auto_painter

Recently, realistic image generation using deep neural networks has become a hot topic in machine learning and computer vision. Such an image can be generated at pixel level by learning from a large collection of images. Learning to generate colorful cartoon images from black-and-white sketches is not only an interesting research problem, but also a useful application in digital entertainment. In this paper, we investigate the sketch-to-image synthesis problem by using conditional generative adversarial networks (cGAN). We propose a model called auto-painter which can automatically generate compatible colors given a sketch. Wasserstein distance is used in training cGAN to overcome model collapse and enable the model converged much better. The new model is not only capable of painting hand-draw sketch with compatible colors, but also allowing users to indicate preferred colors. Experimental results on different sketch datasets show that the auto-painter performs better than other existing image-to-image methods.
Python
132
star
5

EMM-for-stock-prediction

We propose a model to analyze sentiment of online stock forum and use the information to predict stock volatility in the Chinese market. By generating a sentimental dictionary, we analyze the sentimental tendencies of each post as sentiment indicators. Such sentimental information will be fused with market data for prediction based on Recurrent Neural Networks (RNNs). We manually labeled the sentiment of forum post and make the data public available for research. Empirical evidence shows that 8 of the 10 stocks perform better with sentimental indicators.
Python
62
star
6

Auto_painter_demo

The code of building a web demo for Auto_painter
JavaScript
27
star
7

SSIW

The code of 'The devil is in the labels: Semantic segmentation from sentences'.
Python
13
star
8

inceptionV2_finetune

Fine-tuning of inceptionV2 on CUB-200 Birds dataset in Tensorflow
Python
9
star
9

stock_predict

This project predicts stock trends on the basis of online user comments and LSTM
Python
5
star
10

colorization

reading note
3
star
11

horseSeg

raw_code
Python
1
star
12

dvn

dvn for semantic segmentation
1
star