• Stars
    star
    911
  • Rank 50,145 (Top 1.0 %)
  • Language
    C++
  • Created about 3 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[CoRL 21'] TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo

TANDEM: Tracking and Dense Mapping
in Real-time using Deep Multi-view Stereo

Lukas Koestler1* โ€ƒโ€ƒ Nan Yang1,2*,โ€  โ€ƒโ€ƒ Niclas Zeller2,3 โ€ƒโ€ƒ Daniel Cremers1,2

*equal contributionโ€ƒโ€ƒโ€ƒ โ€ corresponding author

1Technical University of Munichโ€ƒโ€ƒโ€ƒ 2Artisense
3Karlsruhe University of Applied Sciences

Conference on Robot Learning (CoRL) 2021, London, UK

3DV 2021 Best Demo Award

arXiv | Video | OpenReview | Project Page

Code and Data

  • ๐Ÿ“ฃ C++ code released before Christmas! Please check tandem/.
  • ๐Ÿ“ฃ CVA-MVSNet released! Please check cva_mvsnet/.
  • ๐Ÿ“ฃ Replica training data released! Please check replica/.
  • Minor improvements throughout January. Contributions are highly welcomed!
  • Release of the ScanNet-trained model
  • Docker image for TANDEM. Contributions are highly welcomed!

Abstract

In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Finally, the predicted depth maps are fused into a consistent global map represented as a truncated signed distance function (TSDF) voxel grid. Our experimental results show that TANDEM outperforms other state-of-the-art traditional and learning-based monocular visual odometry (VO) methods in terms of camera tracking. Moreover, TANDEM shows state-of-the-art real-time 3D reconstruction performance.

Poster

More Repositories

1

lsd_slam

LSD-SLAM
C++
2,486
star
2

LDSO

DSO with SIM(3) pose graph optimization and loop closure
C++
653
star
3

dvo_slam

Dense Visual Odometry and SLAM
C++
607
star
4

fastfusion

Volumetric 3D Mapping in Real-Time on a CPU
C++
543
star
5

online_photometric_calibration

Implementation of online photometric calibration (https://vision.in.tum.de/research/vslam/photometric-calibration)
C++
306
star
6

mono_dataset_code

Code for Monocular Visual Odometry Dataset - https://vision.cs.tum.edu/data/datasets/mono-dataset
C++
261
star
7

tum_ardrone

Repository for the tum_ardrone ROS package, implementing autonomous flight with PTAM-based visual navigation for the Parrot AR.Drone.
C++
221
star
8

dvo

Dense Visual Odometry
C++
148
star
9

fusenet

This repository is the official release of the code for the following paper "FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture" which is published at the 13th Asian Conference on Computer Vision (ACCV 2016).
C++
126
star
10

pnec

[CVPR 2022] README.md The Probabilistic Normal Epipolar Constraint for Frame-To-Frame Rotation Optimization under Uncertain Feature Positions
C++
117
star
11

captcha_recognition

Python
71
star
12

intrinsic-neural-fields

[ECCV '22] Intrinsic Neural Fields: Learning Functions on Manifolds
Jupyter Notebook
66
star
13

dbatk

Distributed Bundle Adjustment Toolkit
59
star
14

fastms

Real-Time Minimization of the Piecewise Smooth Mumford-Shah Functional
C++
57
star
15

ardrone_autonomy

This is a slightly modified version of the official ardrone_autonomy package, which You can find here: https://github.com/AutonomyLab/ardrone_autonomy
C
53
star
16

learn_prox_ops

Implementation of "Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems"
Python
43
star
17

tum_simulator

C++
40
star
18

prost

A fast and flexible convex optimization framework based on proximal splitting
C++
35
star
19

afs

Automatic Feature Selection
C++
31
star
20

rgbd_scribble_benchmark

RGB-D Scribble-based Segmentation Benchmark
Python
26
star
21

autonavx_ardrone

Code for AR.Drone Exercises
C++
24
star
22

autonavx_web

interactive exercises for AUTONAVx course
JavaScript
24
star
23

sublabel_relax

Code for sublabel-accurate multi-labeling papers (published at CVPR '16, ECCV '16)
C++
20
star
24

csd_lmnn

Combined Spectral Descriptors and LMNN for non-rigid 3D shape retrieval
MATLAB
19
star
25

rgbd_demo

Simple ROS demo for processing RGB-D data
C++
17
star
26

mem

Masked Event Modeling: Self-Supervised Pretraining for Event Cameras (WACV '24)
Python
15
star
27

kfusion_ros

ROS integration for kfusion
C++
11
star
28

openni2_camera

OpenNI2 camera node for ROS
C++
9
star
29

articulation

articulation models
C++
6
star
30

nnascg

Source code for experiments in paper "Deriving Neural Network Design and Learning from the Probabilistic Framework of Chain Graphs" by Yuesong Shen and Daniel Cremers.
Python
4
star
31

lgm

Implementation of Layered Graphical Model with demo code
Python
4
star
32

dca

Source code for the NeurIPS 2022 paper "Deep Combinatorial Aggregation"
Python
4
star
33

flbo

2
star
34

hierahyp

1
star