RoutedFusion: Learning Real-time Depth Map Fusion

This is the official and improved implementation of the CVPR 2020 submission RoutedFusion: Learning Real-time Depth Map Fusion.

RoutedFusion is a real-time capable depth map fusion method that leverages machine learning for fusing noisy and outlier-contaminated depth maps. It consists of two neural networks components: 1) the depth routing network that performs a 2D prepocessing of the depth maps estimating a de-noised depth map as well as corresponding confidence map. 2) a depth fusion network that predicts optimal updates to the scene representation given a canonical view of the current state of the scene representation as well as the new measurement and confidence map.

If you find our code or paper useful, please consider citing

@InProceedings{Weder_2020_CVPR,
  author = {Weder, Silvan and Sch\"onberger, Johannes L. and Pollefeys, Marc and Oswald, Martin R.},
  title = {RoutedFusion: Learning Real-Time Depth Map Fusion},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2020}
}

Prior to using the source code in a commercial application, please contact the authors.

Usage

Below you find instructions on how to use RoutedFusion as a standalone depth map fusion pipeline for training and evaluation.

Prerequisites

For using RoutedFusion, you need to have the following installed on you machine:

1. docker (https://www.docker.com/)
2. nvidia-docker (https://github.com/NVIDIA/nvidia-docker)

Data Preparation

The models are trained on the ShapeNet v1 dataset. Therefore, the data needs to be downloaded and perpared using mesh-fusion.

Prerequisites

1. Download ShapeNet v1 from www.shapenet.org and unzip files
2. Download binvox from https://www.patrickmin.com/binvox/, adjust permissions and move it to /usr/bin/
3. Our Docker or conda environment

Install Data Generation

Run the installation script scripts/install_data_generation.sh

Generate Shapenet Data

Run the data generation script scripts/generate_shapenet_data.sh

bash scripts/generate_shapenet_data.sh $PATH_TO_SHAPENET $PATH_TO_GENERATE_TO

Installation

There are two possible ways of installing RoutedFusion. The recommended way is to use Docker. Alternatively, you can also use a conda environment.

Clone the repo

git clone https://github.com/weders/RoutedFusion.git
git submodule update --init --recursive

Build the docker image

docker build . -t routed-fusion

Start and enter the container from the image

docker run -v $PATH_TO_YOUR_PREPROCESSED_DATA:/data -v $PATH_TO_SAVE_EXPERIMENTS:/experiments --gpus all  -it routed-fusion:latest

Alternatively, create the Anaconda environment

conda env create -f environment.yml
conda activate routed-fusion
bash scripts/install_docker.sh

Training

Once you are in the docker container you can train RoutedFusion. First, you can train the routing network. Secondly, you can train the fusion network.

Train Routing Network

python train_routing.py --config configs/routing/shapenet.noise.005.yaml

Train Fusion Network

without routing

python train_fusion.py --config configs/fusion/shapenet.noise.005.yaml

with routing

python train_fusion.py --config configs/fusion/shapenet.noise.005.yaml --routing-model $PATH_TO_YOUR_ROUTING_MODEL

Change Data Configuration For training RoutedFusion with ShapeNet using a different artificial noise model, you can simply change the input key in the config file and add the corresponding noise model to the dataset class.

Testing

You can test RoutedFusion using either the pretrained models or your own model. Furthermore, you need to define a test config specifying the test data.

test our full pretrained model

python test_fusion.py --experiment pretrained_models/fusion/shapenet_noise_005 --test configs/tests/shapenet.routed.noise.005.yaml

test your own model

python test_fusion.py --experiment $PATH_TO_YOUR_EXPERIMENT --test configs/tests/shapenet.routed.noise.005.yaml

Train and test RoutedFusion on your own data

In order to train and/or test RoutedFusion on your own data, you need to add a new dataset class with the same interface as shown in the ShapeNet dataset class. You need to make sure that all keys are available. Moreover, you need to write your test configuration file and you are ready to go.