fregu856/evaluating_bdl fregu856/evaluating_bdl

  • Stars
    star
    129
  • Rank 279,262 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created over 5 years ago
  • Updated over 4 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
fregu856/evaluating_bdl
github

fregu856

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Official implementation of "Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision", CVPR Workshops 2020.

evaluating_bdl

overview image

Official implementation (PyTorch) of the paper:
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision, CVPR Workshops 2020 [arXiv] [project].
Fredrik K. Gustafsson, Martin Danelljan, Thomas B. SchΓΆn.

We propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. It is specifically designed to test the robustness required in real-world computer vision applications. We also apply our proposed framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates.

Youtube video with qualitative results:
demo video with qualitative results

If you find this work useful, please consider citing:

@inproceedings{gustafsson2020evaluating,
  title={Evaluating scalable Bayesian deep learning methods for robust computer vision},
  author={Gustafsson, Fredrik K and Danelljan, Martin and Sch{\"o}n, Thomas B},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year={2020}
}

Acknowledgements

Index

Usage

The code has been tested on Ubuntu 16.04. Docker images are provided (see below).

depthCompletion

  • $ sudo docker pull fregu856/evaluating_bdl:pytorch_pytorch_0.4_cuda9_cudnn7_evaluating_bdl
  • Create start_docker_image_toyProblems_depthCompletion.sh containing (My username on the server is fregu482, i.e., my home folder is /home/fregu482. You will have to modify this accordingly):
#!/bin/bash

# DEFAULT VALUES
GPUIDS="0"
NAME="toyProblems_depthCompletion_GPU"

NV_GPU="$GPUIDS" nvidia-docker run -it --rm --shm-size 12G \
        -p 5700:5700\
        --name "$NAME""0" \
        -v /home/fregu482:/root/ \
        fregu856/evaluating_bdl:pytorch_pytorch_0.4_cuda9_cudnn7_evaluating_bdl bash

  • (Inside the image, /root/ will now be mapped to /home/fregu482, i.e., $ cd -- takes you to the regular home folder)

  • (To create more containers, change the lines GPUIDS="0", --name "$NAME""0" and -p 5700:5700)

  • General Docker usage:

    • To start the image:
      • $ sudo sh start_docker_image_toyProblems_depthCompletion.sh
    • To commit changes to the image:
      • Open a new terminal window.
      • $ sudo docker commit toyProblems_depthCompletion_GPU0 fregu856/evaluating_bdl:pytorch_pytorch_0.4_cuda9_cudnn7_evaluating_bdl
    • To exit the image without killing running code:
      • Ctrl + P + Q
    • To get back into a running image:
      • $ sudo docker attach toyProblems_depthCompletion_GPU0

  • Download the KITTI depth completion dataset (data_depth_annotated.zip, data_depth_selection.zip and data_depth_velodyne.zip) and place it in /root/data/kitti_depth (/root/data/kitti_depth should contain the folders train, val and depth_selection).

  • Create /root/data/kitti_raw and download the KITTI raw dataset using download_kitti_raw.py.

  • Create /root/data/kitti_rgb. For each folder in /root/data/kitti_depth/train (e.g. 2011_09_26_drive_0001_sync), copy the corresponding folder in /root/data/kitti_raw and place it in /root/data/kitti_rgb/train.

  • Download the virtual KITTI dataset (vkitti_1.3.1_depthgt.tar and vkitti_1.3.1_rgb.tar) and place in /root/data/virtualkitti (/root/data/virtualkitti should contain the folders vkitti_1.3.1_depthgt and vkitti_1.3.1_rgb).

  • Example usage:

$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/depthCompletion/ensembling_train_virtual.py

segmentation

  • $ sudo docker pull fregu856/evaluating_bdl:rainbowsecret_pytorch04_20180905_evaluating_bdl
  • Create start_docker_image_segmentation.sh containing (My username on the server is fregu482, i.e., my home folder is /home/fregu482. You will have to modify this accordingly):
#!/bin/bash

# DEFAULT VALUES
GPUIDS="0,1"
NAME="segmentation_GPU"

NV_GPU="$GPUIDS" nvidia-docker run -it --rm --shm-size 12G \
        -p 5900:5900 \
        --name "$NAME""01" \
        -v /home/fregu482:/home/ \
        fregu856/evaluating_bdl:rainbowsecret_pytorch04_20180905_evaluating_bdl bash

  • (Inside the image, /home/ will now be mapped to /home/fregu482, i.e., $ cd home takes you to the regular home folder)

  • (To create more containers, change the lines GPUIDS="0,1", --name "$NAME""01" and -p 5900:5900)

  • General Docker usage:

    • To start the image:
      • $ sudo sh start_docker_image_segmentation.sh
    • To commit changes to the image:
      • Open a new terminal window.
      • $ sudo docker commit segmentation_GPU01 fregu856/evaluating_bdl:rainbowsecret_pytorch04_20180905_evaluating_bdl
    • To exit the image without killing running code:
      • Ctrl + P + Q
    • To get back into a running image:
      • $ sudo docker attach segmentation_GPU01

  • Download resnet101-imagenet.pth from here and place it in segmentation.

  • Download the Cityscapes dataset and place it in /home/data/cityscapes (/home/data/cityscapes should contain the folders leftImg8bit and gtFine).

  • Download the Synscapes dataset and place it in /home/data/synscapes (/home/data/synscapes should contain the folder img, which in turn should contain the folders rgb-2k and class).

  • Run segmentation/utils/preprocess_synscapes.py (This will, among other things, create /home/data/synscapes_meta/train_img_ids.pkl and /home/data/synscapes_meta/val_img_ids.pkl by randomly selecting subsets of examples. The ones used in the paper are found in segmentation/lists/synscapes).

  • Example usage:

$ sudo sh start_docker_image_segmentation.sh
$ cd home
$ /root/miniconda3/bin/python evaluating_bdl/segmentation/ensembling_train_syn.py

toyRegression

  • Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyRegression/Ensemble-Adam/train.py

toyClassification

  • Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyClassification/Ensemble-Adam/train.py

Documentation

Documentation/depthCompletion

  • Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/depthCompletion/ensembling_train_virtual.py

  • criterion.py: Definitions of losses and metrics.

  • datasets.py: Definitions of datasets, for KITTI depth completion (KITTI) and virtualKITTI.

  • model.py: Definition of the CNN.

  • model_mcdropout.py: Definition of the CNN, with inserted dropout layers.

  • %%%%%

  • ensembling_train.py: Code for training M model.py models, on KITTI train.

  • ensembling_train_virtual.py: As above, but on virtualKITTI train.

  • ensembling_eval.py: Computes the loss and RMSE for a trained ensemble, on KITTI val. Also creates visualization images of the input data, ground truth, prediction and the estimated uncertainty.

  • ensembling_eval_virtual.py: As above, but on virtualKITTI val.

  • ensembling_eval_auce.py: Computes the AUCE (mean +- std) for M = [1, 2, 4, 8, 16, 32] on KITTI val, based on a total of 33 trained ensemble members. Also creates calibration plots.

  • ensembling_eval_auce_virtual.py: As above, but on virtualKITTI val.

  • ensembling_eval_ause.py: Computes the AUSE (mean +- std) for M = [1, 2, 4, 8, 16, 32] on KITTI val, based on a total of 33 trained ensemble members. Also creates sparsification plots and sparsification error curves.

  • ensembling_eval_ause_virtual.py: As above, but on virtualKITTI val.

  • ensembling_eval_seq.py: Creates visualization videos (input data, ground truth, prediction and the estimated uncertainty) for a trained ensemble, on all sequences in KITTI val.

  • ensembling_eval_seq_virtual.py: As above, but on all sequences in virtualKITTI val.

  • %%%%%

  • mcdropout_train.py: Code for training M model_mcdropout.py models, on KITTI train.

  • mcdropout_train_virtual.py: As above, but on virtualKITTI train.

  • mcdropout_eval.py: Computes the loss and RMSE for a trained MC-dropout model with M forward passes, on KITTI val. Also creates visualization images of the input data, ground truth, prediction and the estimated uncertainty.

  • mcdropout_eval_virtual.py: As above, but on virtualKITTI val.

  • mcdropout_eval_auce.py: Computes the AUCE (mean +- std) for M = [1, 2, 4, 8, 16, 32] forward passes on KITTI val, based on a total of 16 trained MC-dropout models. Also creates calibration plots.

  • mcdropout_eval_auce_virtual.py: As above, but on virtualKITTI val.

  • mcdropout_eval_ause.py: Computes the AUSE (mean +- std) for M = [1, 2, 4, 8, 16, 32] forward passes on KITTI val, based on a total of 16 trained MC-dropout models. Also creates sparsification plots and sparsification error curves.

  • mcdropout_eval_ause_virtual.py: As above, but on virtualKITTI val.

  • mcdropout_eval_seq.py: Creates visualization videos (input data, ground truth, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, on all sequences in KITTI val.

  • mcdropout_eval_seq_virtual.py: As above, but on all sequences in virtualKITTI val.

Documentation/segmentation

  • Example usage:
$ sudo sh start_docker_image_segmentation.sh
$ cd home
$ /root/miniconda3/bin/python evaluating_bdl/segmentation/ensembling_train_syn.py

  • models:

      • model.py: Definition of the CNN.
      • model_mcdropout.py: Definition of the CNN, with inserted dropout layers.
      • aspp.py: Definition of the ASPP module.
      • resnet_block.py: Definition of a ResNet block.

  • utils:

      • criterion.py: Definition of the cross-entropy loss.
      • preprocess_synscapes.py: Creates the Synscapes train (val) dataset by randomly selecting a subset of 2975 (500) examples, and resizes the labels to 1024 x 2048.
      • utils.py: Helper functions for evaluation and visualization.

  • datasets.py: Definitions of datasets, for Cityscapes and Synscapes.

  • %%%%%

  • ensembling_train.py: Code for training M model.py models, on Cityscapes train.

  • ensembling_train_syn.py: As above, but on Synscapes train.

  • ensembling_eval.py: Computes the mIoU for a trained ensemble, on Cityscapes val. Also creates visualization images of the input image, ground truth, prediction and the estimated uncertainty.

  • ensembling_eval_syn.py: As above, but on Synscapes val.

  • ensembling_eval_ause_ece.py: Computes the AUSE (mean +- std) and ECE (mean +- std) for M = [1, 2, 4, 8, 16] on Cityscapes val, based on a total of 26 trained ensemble members. Also creates sparsification plots, sparsification error curves and reliability diagrams.

  • ensembling_eval_ause_ece_syn.py: As above, but on Synscapes val.

  • ensembling_eval_seq.py: Creates visualization videos (input image, prediction and the estimated uncertainty) for a trained ensemble, on the three demo sequences in Cityscapes.

  • ensembling_eval_seq_syn.py: Creates a visualization video (input image, ground truth, prediction and the estimated uncertainty) for a trained ensemble, showing the 30 first images in Synscapes val.

  • %%%%%

  • mcdropout_train.py: Code for training M model_mcdropout.py models, on Cityscapes train.

  • mcdropout_train_syn.py: As above, but on Synscapes train.

  • mcdropout_eval.py: Computes the mIoU for a trained MC-dropout model with M forward passes, on Cityscapes val. Also creates visualization images of the input image, ground truth, prediction and the estimated uncertainty.

  • mcdropout_eval_syn.py: As above, but on Synscapes val.

  • mcdropout_eval_ause_ece.py: Computes the AUSE (mean +- std) and ECE (mean +- std) for M = [1, 2, 4, 8, 16] forward passes on Cityscapes val, based on a total of 8 trained MC-dropout models. Also creates sparsification plots, sparsification error curves and reliability diagrams.

  • mcdropout_eval_ause_ece_syn.py: As above, but on Synscapes val.

  • mcdropout_eval_seq.py: Creates visualization videos (input image, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, on the three demo sequences in Cityscapes.

  • mcdropout_eval_seq_syn.py: Creates a visualization video (input image, ground truth, prediction and the estimated uncertainty) for a trained MC-dropout model with M forward passes, showing the 30 first images in Synscapes val.

Documentation/toyRegression

  • Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyRegression/Ensemble-Adam/train.py

  • Ensemble-Adam:

    • Ensembling by minimizing the MLE objective using Adam and random initialization.
      • datasets.py: Definition of the training dataset.
      • model.py: Definition of the feed-forward neural network.
      • train.py: Code for training M models.
      • eval.py: Creates a plot of the obtained predictive distribution and the HMC "ground truth" predictive distribution, for a set value of M. Also creates histograms for the model parameters.
      • eval_plots.py: Creates plots of the obtained predictive distributions for different values of M.
      • eval_kl_div.py: Computes the KL divergence between the obtained predictive distribution and the HMC "ground truth", for different values of M.

  • Ensemble-MAP-Adam:

      • Ensembling by minimizing the MAP objective using Adam and random initialization.

  • Ensemble-MAP-Adam-Fixed:

      • Ensembling by minimizing the MAP objective using Adam and NO random initialization.

  • Ensemble-MAP-SGD:

      • Ensembling by minimizing the MAP objective using SGD and random initialization.

  • Ensemble-MAP-SGDMOM:

      • Ensembling by minimizing the MAP objective using SGDMOM and random initialization.

  • MC-Dropout-MAP-02-Adam:

      • MC-dropout by minimizing the MAP objective using Adam, p=0.2.

  • MC-Dropout-MAP-02-SGD

      • MC-dropout by minimizing the MAP objective using SGD, p=0.2.

  • MC-Dropout-MAP-02-SGDMOM:

      • MC-dropout by minimizing the MAP objective using SGDMOM, p=0.2.

  • SGLD-256:

      • Implementation of SGLD, trained for 256 times longer than each member of an ensemble.

  • SGLD-64:

      • Implementation of SGLD, trained for 64 times longer than each member of an ensemble..

  • SGHMC-256:

      • Implementation of SGHMC, trained for 256 times longer than each member of an ensemble.

  • SGHMC-64:

      • Implementation of SGHMC, trained for 64 times longer than each member of an ensemble.

  • HMC:

      • Implementation of HMC using Pyro.

  • Deterministic:

      • Implementation of a fully deterministic model, i.e., direct regression.

Documentation/toyClassification

  • Example usage:
$ sudo sh start_docker_image_toyProblems_depthCompletion.sh
$ cd --
$ python evaluating_bdl/toyClassification/Ensemble-Adam/train.py

  • Ensemble-Adam:

    • Ensembling by minimizing the MLE objective using Adam and random initialization.
      • datasets.py: Definition of the training dataset.
      • model.py: Definition of the feed-forward neural network.
      • train.py: Code for training M models.
      • eval.py: Creates a plot of the obtained predictive distribution and the HMC "ground truth" predictive distribution, for a set value of M. Also creates histograms for the model parameters.
      • eval_plots.py: Creates plots of the obtained predictive distributions for different values of M.
      • eval_kl_div.py: Computes the KL divergence between the obtained predictive distribution and the HMC "ground truth", for different values of M.

  • Ensemble-Adam-Fixed:

      • Ensembling by minimizing the MLE objective using Adam and NO random initialization.

  • Ensemble-MAP-Adam:

      • Ensembling by minimizing the MAP objective using Adam and random initialization.

  • Ensemble-MAP-SGD:

      • Ensembling by minimizing the MAP objective using SGD and random initialization.

  • Ensemble-MAP-SGDMOM:

      • Ensembling by minimizing the MAP objective using SGDMOM and random initialization.

  • MC-Dropout-MAP-01-Adam:

      • MC-dropout by minimizing the MAP objective using Adam, p=0.1.

  • MC-Dropout-MAP-02-SGD

      • MC-dropout by minimizing the MAP objective using SGD, p=0.2.

  • MC-Dropout-MAP-02-SGDMOM:

      • MC-dropout by minimizing the MAP objective using SGDMOM, p=0.2.

  • SGLD-256:

      • Implementation of SGLD, trained for 256 times longer than each member of an ensemble.

  • SGLD-64:

      • Implementation of SGLD, trained for 64 times longer than each member of an ensemble..

  • SGHMC-256:

      • Implementation of SGHMC, trained for 256 times longer than each member of an ensemble.

  • SGHMC-64:

      • Implementation of SGHMC, trained for 64 times longer than each member of an ensemble.

  • HMC:

      • Implementation of HMC using Pyro.

Pretrained models

More Repositories

1

deeplabv3

PyTorch implementation of DeepLabV3, trained on the Cityscapes dataset.
Python
760
star
2

papers

I categorize, annotate and write comments for all research papers I read (410+ papers since 2018).
398
star
3

3DOD_thesis

3D Object Detection for Autonomous Driving in PyTorch, trained on the KITTI dataset.
Python
282
star
4

segmentation

TensorFlow implementation of ENet, trained on the Cityscapes dataset.
Python
244
star
5

2D_detection

TensorFlow implementation of SqueezeDet, trained on the KITTI dataset.
Python
130
star
6

ebms_regression

Official implementation of "Energy-Based Models for Deep Probabilistic Regression" (ECCV 2020) and "How to Train Your Energy-Based Model for Regression" (BMVC 2020).
Jupyter Notebook
94
star
7

CS224n_project

Neural Image Captioning in TensorFlow.
Jupyter Notebook
65
star
8

ebms_3dod

Official implementation of "Accurate 3D Object Detection using Energy-Based Models", CVPR Workshops 2021.
Python
59
star
9

regression_uncertainty

Official implementation of "How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?", TMLR 2023.
Python
18
star
10

ebms_proposals

Official implementation of "Learning Proposals for Practical Energy-Based Regression", AISTATS 2022.
Python
11
star
11

Sommarprojekt16

Python
10
star
12

Summer17

Makefile
6
star
13

skalman

Makefile
3
star
14

fregu856.github.io

JavaScript
3
star
15

CS-229

Python
3
star
16

TSRT10

Makefile
2
star