• Stars
    star
    574
  • Rank 77,739 (Top 2 %)
  • Language
    Python
  • License
    GNU General Publi...
  • Created over 7 years ago
  • Updated over 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

DeMoN: Depth and Motion Network

DeMoN: Depth and Motion Network

License

DeMoN is "A computer algorithm for reconstructing a scene from two projections"1. The network estimates the depth and relative camera motion for pairs of images; it addresses the important two view case in structure from motion.

Teaser

If you use this code for research please cite:

@InProceedings{UZUMIDB17,
  author       = "B. Ummenhofer and H. Zhou and J. Uhrig and N. Mayer and E. Ilg and A. Dosovitskiy and T. Brox",
  title        = "DeMoN: Depth and Motion Network for Learning Monocular Stereo",
  booktitle    = "IEEE Conference on Computer Vision and Pattern Recognition (CVPR)",
  month        = " ",
  year         = "2017",
  url          = "http://lmb.informatik.uni-freiburg.de//Publications/2017/UZUMIDB17"
}

See the project website for the paper and other material.

1 This is the title of H. C. Longuet-Higgins paper from 1981, which perfectly describes what our method does. DeMoN shows that complex geometric relations can be learnt by a ConvNet.

Requirements

Building and using requires the following libraries and programs

tensorflow 1.4.0
cmake 3.7.1
python 3.5
cuda 8.0.61 (required for gpu support)
VTK 7.1 with python3 interface (required for visualizing point clouds)

The versions match the configuration we have tested on an ubuntu 16.04 system. DeMoN can work with other versions of the aforementioned dependencies, e.g. tensorflow 1.3, but this is not well tested.

The binary package from vtk.org does not come with a python3 interface. To enable python3 support VTK needs to be built from source. Alternatively, there are also VTK packages with python3 support available in Anaconda via the conda package manager.

The network also depends on our lmbspecialops library which is included as a submodule.

Build instructions

The following describes how to install tensorflow and demon into a new virtualenv and run the inference example. We will use pew (pip3 install pew) to manage a new virtualenv named demon_venv in the following:

# create virtualenv
pew new demon_venv

The following commands all run inside the virtualenv:

# install python module dependencies
pip3 install tensorflow-gpu # or 'tensorflow' without gpu support
pip3 install pillow # for reading images
pip3 install matplotlib # required for visualizing depth maps
pip3 install Cython # required for visualizing point clouds
# clone repo with submodules
git clone --recursive https://github.com/lmb-freiburg/demon.git

# build lmbspecialops
DEMON_DIR=$PWD/demon
mkdir $DEMON_DIR/lmbspecialops/build
cd $DEMON_DIR/lmbspecialops/build
cmake .. # add '-DBUILD_WITH_CUDA=OFF' to build without gpu support
# (optional) run 'ccmake .' here to adjust settings for gpu code generation
make
pew add $DEMON_DIR/lmbspecialops/python # add to python path

# download weights
cd $DEMON_DIR/weights
./download_weights.sh

# run example
cd $DEMON_DIR/examples
python3 example.py # opens a window with the depth map (and the point cloud if vtk is available)

Data reader op & evaluation

The data reader op and the evaluation code have additional dependencies. The code for the data reader is in the multivih5datareaderop directory. See the corresponding readme for more details.

For the evaluation see the example examples/evaluation.py. The evaluation code requires the following additional python3 packages, which can be installed with pip:

h5py
minieigen
pandas
scipy
scikit-image
xarray

Note that the evaluation code also depends on the data reader op.

Training code

Instructions for training a clean tensorflow version of DeMoN are here. Note that the tensorflow training code and model are work in progress and are not identical to the original Caffe version.

Datasets

Download scripts for training and testing are located in the datasets subdirectory. Note that due to a bug that some of the dataset files with the prefix rgbd did contain some samples from the test set. The affected files have been replaced and now have the prefix rgbd_bugfix. MD5 checksums for all files can be found in the file traindata.md5.

Docker build

Ensure Docker is installed on your system, and that the default Docker runtime is Nvidia:

{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

Then issue the Docker build command:

$ docker build . -t demon

To visualize the example:

$ docker run --gpus all -it -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix:ro demon

License

DeMoN is under the GNU General Public License v3.0

More Repositories

1

flownet2

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
C++
1,004
star
2

hand3d

Network estimating 3D Handpose from single color images
Python
801
star
3

freihand

A dataset for estimation of hand pose and shape from single color images.
Python
382
star
4

deeptam

DeepTAM: Deep Tracking and Mapping https://lmb.informatik.uni-freiburg.de/people/zhouh/deeptam/
Python
233
star
5

mv3d

Multi-view 3D Models from Single Images with a Convolutional Network
Python
214
star
6

rgbd-pose3d

3D Human Pose Estimation in RGBD Images for Robotic Task Learning
Python
198
star
7

flownet2-docker

Dockerfile and runscripts for FlowNet 2.0 (estimation of optical flow)
Shell
158
star
8

netdef_models

Repository for different network models related to flow/disparity (ECCV 18)
Python
157
star
9

ogn

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs
C++
155
star
10

orion

ORION: Orientation-boosted Voxel Nets for 3D Object Recognition
MATLAB
111
star
11

what3d

What Do Single-view 3D Reconstruction Networks Learn?
Python
98
star
12

dispnet-flownet-docker

Dockerfile and runscripts for DispNet and FlowNet1 (estimation of disparity and optical flow)
Shell
87
star
13

Unet-Segmentation

The U-Net Segmentation plugin for Fiji (ImageJ)
Java
87
star
14

robustmvd

Repository for the Robust Multi-View Depth Benchmark
Python
74
star
15

contra-hand

Code in conjunction with the publication 'Contrastive Representation Learning for Hand Shape Estimation'
Python
53
star
16

Multimodal-Future-Prediction

The official repository for the CVPR 2019 paper "Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction"
Python
47
star
17

lmbspecialops

A collection of tensorflow ops
C++
46
star
18

FLN-EPN-RPN

This repository contains the source code of the CVPR 2020 paper: "Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View with a Reachability Prior"
Python
32
star
19

flow_rl

Python
28
star
20

netdef-docker

DispNet3, FlowNet3, FlowNetH, SceneFlowNet -- in Docker
Shell
28
star
21

caffe-unet-docker

The U-Net Segmentation server (caffe_unet) for Docker
Shell
27
star
22

Contrastive-Future-Trajectory-Prediction

The official repository of the ICCV paper "On Exposing the Challenging Long Tail in Future Prediction of Traffic Actors"
Python
25
star
23

locov

Localized Vision-Language Matching for Open-vocabulary Object Detection
Python
19
star
24

unsup-car-dataset

Unsupervised Generation of a Viewpoint Annotated Car Dataset from Videos
MATLAB
19
star
25

FreiPose-docker

FreiPose: A Deep Learning Framework for Precise Animal Motion Capture in 3D Spaces
Dockerfile
18
star
26

optical-flow-2d-data-generation

Caffe(v1)-compatible codebase to generate optical flow training data on-the-fly; used for the IJCV 2018 paper "What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?" (http://dx.doi.org/10.1007/s11263-018-1082-6)
C++
18
star
27

autodispnet

Code for AutoDispNet (ICCV 2019)
Python
17
star
28

cv-exercises

Python
15
star
29

spr-exercises

Jupyter Notebook
12
star
30

td-or-not-td

Code for the paper "TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning", Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun and Thomas Brox, ICLR 2018
Python
12
star
31

sf2se3

Repository for SF2SE3: Clustering Scene Flow into SE(3)-Motions via Proposal and Selection
Python
10
star
32

ovqa

Python
10
star
33

understanding_flow_robustness

Official repository for "Towards Understanding Adversarial Robustness of Optical Flow Networks" (CVPR 2022)
Python
9
star
34

neural-point-cloud-diffusion

Official repository for "Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation"
Python
9
star
35

ldce

Official repository for "Latent Diffusion Counterfactual Explanations"
Python
9
star
36

PreFAct

Code and Models for the paper "Learning Representations for Predicting Future Activities"
8
star
37

ROS-packages

A collection of ROS packages for LMB software; DispNet(1+3), FlowNet2, etc.
C++
7
star
38

FreiPose

C++
7
star
39

diffusion-for-ood

Official repository for "Diffusion for Out-of-Distribution Detection on Road Scenes and Beyond". Coming soon.
Python
5
star
40

tfutils

tfutils is a set of tools for training networks with tensorflow
Python
5
star
41

FreiCalib

C++
5
star
42

netdef_slim

A python wrapper for tf to ease creation of network definitions.
Python
4
star
43

iRoCS-Toolbox

n-D Image Analysis libraries and tools
C++
4
star
44

rohl

Python
3
star
45

RecordTool

Python
2
star
46

tree-planting

Official repository for "Climate-sensitive Urban Planning Through Optimization of Tree Placements"
Python
2
star
47

ade-ood

Official repo for the ADE-OoD benchmark.
Python
1
star