• Stars
    star
    347
  • Rank 122,141 (Top 3 %)
  • Language Cuda
  • License
    GNU General Publi...
  • Created over 6 years ago
  • Updated about 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This is a monocular dense mapping system corresponding to IROS 2018 "Quadtree-accelerated Real-time Monocular Dense Mapping"

QuadtreeMapping

A Real-time Monocular Dense Mapping System

This is a monocular dense mapping system following the IROS 2018 paper Quadtree-accelerated Real-time Monocular Dense Mapping, Kaixuan Wang, Wenchao Ding, Shaojie Shen.

Given a localized monocular camera, the system can generate dense depth maps in real-time using portable devices. The generated depth maps can be used to reconstruct the environment or be used for UAV autonomous flight. An example of real-time reconstruction is mapping example

Red line is the camera trajectory.

A video can be used to illustrate the pipeline and the performance of our system:

video

We would like to thank rpg_open_remode for their open source work. The project inspires us, and the system architecture helps we build QuadtreeMapping.

Please note that, in the system, the depth value is defined as Euclidean distance instead of z value. For example, if the point is (x, y, z) in camera coordinate, the depth value is

This branch uses opencv totally on CPU. If you need to speed up the image undistort step by using OpenCV with CUDA, please checkout to master branch.

1.0 Prerequisites

  • Ubuntu and ROS

Both Ubuntu 16.04 with ROS Kinect and Ubuntu 14.04 with ROS Indigo are ok.

  • CUDA

The system uses GPU to parallel most of the computation. You don't need a powerful GPU to run the code, but it must be a Nvidia GPU that supports CUDA. We use CUDA 8.0 to run the system. CUDA 9.0 has not been tested yet.

  • OpenCV

The OpenCV that comes with ROS is ok.

2.0 install

Since the GPU device varies from each one to another, the CMakeLists.txt needs to be changed accordingly.

cd ~/catkin_ws/src
git clone https://github.com/HKUST-Aerial-Robotics/open_quadtree_mapping.git
cd open_quadtree_mapping

now find the CMakeLists.txt

First, change the Compute capability in line 11 and line 12 according to your device. The default value is 61 and it works for Nvidia TITAN Xp etc. The compute capability of your device can be found at wikipedia.

After the change of CMakeLists.txt, you can compile the QuadtreeMapping.

cd ~/catkin_ws
catkin_make

3.0 parameters

Before running the system, please take a look at the parameters in the launch/example.launch.

  • cam_width, cam_height, cam_fx, cam_cx, cam_fy, cam_cy, cam_k1, cam_k2, cam_r1, cam_r2 are the camera intrinsic parameters. We use pinhole model.
  • downsample_factor is used to resize the image. The estimated depth maps have size of cam_width*downsample_factor x cam_height*downsample_factor. This factor is useful if you want to run QuadtreeMapping on platforms with limited resources, for example Jetson TX1.
  • semi2dense_ratio is the ratio to control output depth map frequency and must be an integer. If the input camera-pose pairs are 30Hz and you only need 10Hz depth estimation, set semi2dense_ratio to 3. High-frequency camera-pose input works better than a low-frequency input even you want a low-frequency depth estimation.

4.0 run QuadtreeMapping

The input of QuadtreeMapping is synchronized Image (sensor_msgs::Image) and Pose (geometry_msgs::PoseStamped). Make sure the ROS messages are the correct type and the timestamps are the same. Images and poses at different frequencies is ok. For example, the system will filter 30Hz images and 10Hz poses into 10Hz image-pose pairs as input.

4.1 run the example

We provide an example of a hand-held camera navigating in a garden. We provide the link to download the bag. The ego-motion is estimated using VINS-MONO.

To run the example, just

roslaunch open_quadtree_mapping example.launch

and play the bag in another terminal

rosbag play example.bag

The results are published as:

  • /open_quadtree_mapping/depth : estimated depth for each pixel, invalid depth are filled with zeros,
  • /open_quadtree_mapping/color_depth : color-coded depth maps for visualization, invaild depth are red,
  • /open_quadtree_mapping/debug : color-coded depth of pixels before depth interpolation.
  • /open_quadtree_mapping/reference : undistorted intensity image,
  • /open_quadtree_mapping/pointcloud : point cloud from the current undistorted intensity image and the extracted depth map.

4.2 run with other datasets or run live

To run with other data, you can modify the launch file according to your settings. To get good results, a few things need to be noticed:

  • Good ego-motion estimation is required. The ego-motion should be precise and have the metric scale. We recommend using VINS-MONO to estimate the camera motion. Visual odometry systems like ORB-SLAM cannot be directly used unless the scale information is recovered.
  • Rotation is not good for the system. Rotation reduces the number of frames QuadtreeMapping can use to estimate the depth map.
  • A good camera is required. A good choice is an industry camera that has a global shutter and is set to a fixed exposure time. Also, images should have a balanced contrast, too bright or too dark is not good.

5.0 fuse into a global map

Quadtree publishes depth maps and the corresponding intensity images. You can fuse them using the tool you like. We use a modified open chisel for 3D reconstruction and use a GPU-based TSDF to support autonomous flight.

6.0 future update

  • modified open chisel will be open source soon.

More Repositories

1

VINS-Mono

A Robust and Versatile Monocular Visual-Inertial State Estimator
C++
4,967
star
2

VINS-Fusion

An optimization-based multi-sensor state estimator
C++
3,181
star
3

Fast-Planner

A Robust and Efficient Trajectory Planner for Quadrotors
C++
2,433
star
4

A-LOAM

Advanced implementation of LOAM
C++
1,957
star
5

VINS-Mobile

Monocular Visual-Inertial State Estimator on Mobile Phones
C++
1,269
star
6

Teach-Repeat-Replan

Teach-Repeat-Replan: A Complete and Robust System for Aggressive Flight in Complex Environments
C++
951
star
7

GVINS

Tightly coupled GNSS-Visual-Inertial system for locally smooth and globally consistent state estimation in complex environment.
C++
882
star
8

FUEL

An Efficient Framework for Fast UAV Exploration
C++
744
star
9

Stereo-RCNN

Code for 'Stereo R-CNN based 3D Object Detection for Autonomous Driving' (CVPR 2019)
Python
690
star
10

DenseSurfelMapping

This is the open-source version of ICRA 2019 submission "Real-time Scalable Dense Surfel Mapping"
C++
661
star
11

FIESTA

Fast Incremental Euclidean Distance Fields for Online Motion Planning of Aerial Robots
C++
617
star
12

EPSILON

C++
493
star
13

ESVO

This repository maintains the implementation of "Event-based Stereo Visual Odometry".
C++
408
star
14

Btraj

Bezier Trajectory Generation for Autonomous Quadrotor, ICRA 2018
C++
407
star
15

grad_traj_optimization

Gradient-Based Online Safe Trajectory Generator
C++
363
star
16

MonoLaneMapping

Online Monocular Lane Mapping Using Catmull-Rom Spline (IROS 2023)
Python
349
star
17

MVDepthNet

This repository provides PyTorch implementation for 3DV 2018 paper "MVDepthNet: real-time multiview depth estimation neural network"
Python
305
star
18

D2SLAM

$D^2$SLAM: Decentralized and Distributed Collaborative Visual-inertial SLAM System for Aerial Swarm
Jupyter Notebook
277
star
19

OmniNxt

[IROS 2024 Oral] A Fully Open-source and Compact Aerial Robot with Omnidirectional Visual Perception
255
star
20

G3Reg

A fast and robust global registration library for outdoor LiDAR point clouds.
C++
200
star
21

GVINS-Dataset

A dataset containing synchronized visual, inertial and GNSS raw measurements.
C++
197
star
22

Nxt-FC

Mini PX4 for UAV Group
Shell
187
star
23

Omni-swarm

A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swar.
Jupyter Notebook
179
star
24

spatiotemporal_semantic_corridor

Implementation of the paper "Safe Trajectory Generation For Complex Urban Environments Using Spatio-temporal Semantic Corridor".
C++
160
star
25

PredRecon

[ICRA 2023] A Prediction-boosted Planner for Fast and High-quality Autonomous Aerial Reconstruction
C++
156
star
26

FC-Planner

[ICRA 2024 Best UAV Paper Award Finalist] An Efficient Gloabl Planner for Aerial Coverage
C++
155
star
27

eudm_planner

Implementation of the paper "Efficient Uncertainty-aware Decision-making for Automated Driving Using Guided Branching".
C++
139
star
28

mockamap

a simple map generator based on ROS
C++
133
star
29

DSP

Trajectory Prediction with Graph-based Dual-scale Context Fusion
Python
132
star
30

pointcloudTraj

Trajectory generation on point clouds
C++
128
star
31

Pagor

Pyramid Semantic Graph-based Global Point Cloud Registration with Low Overlap (IROS 2023)
C++
127
star
32

Flow-Motion-Depth

This is the project page of the paper "Flow-Motion and Depth Network for Monocular Stereo and Beyond''
Python
114
star
33

gnss_comm

Basic definitions and utility functions for GNSS raw measurement processing
C++
111
star
34

SIMPL

SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving
Python
107
star
35

VINS-kidnap

a place recognition system for VINS-fusion
105
star
36

ublox_driver

A driver for u-blox receiver (ZED-F9P) with ros support
C++
102
star
37

TopoTraj

A robust UAV local planner based on the ICRA2020 paper: Robust Real-time UAV Replanning Using Guided Gradient-based Optimization and Topological Paths
90
star
38

TimeOptimizer

Optimal Time Allocation for Quadrotor Trajectory Generation
C++
83
star
39

AutoTrans

AutoTrans: A Complete Planning and Control Framework for Autonomous UAV Payload Transportation.
C++
76
star
40

LiDAR-Registration-Benchmark

LiDAR-based 3D global registration benchmark.
Python
75
star
41

Pinhole-Fisheye-Mapping

70
star
42

UniQuad

UniQuad: A Unified and Versatile Quadrotor Platform Series for UAV Research and Application
67
star
43

IMPACTOR

Impact-Aware Planning and Control for Aerial Robots with Suspended Payloads
C
67
star
44

SLABIM

An open-sourced SLAM dataset that couples with BIM (Building Information Modeling).
Python
66
star
45

HKUST-ELEC5660-Introduction-to-Aerial-Robotics

Repo for HKUST ELEC5660 Course Notes & Lab Tutorial & Project Docker
C++
57
star
46

EMSGC

This repository maintains the implementation of the paper "Event-based Motion Segmentation withSpatio-Temporal Graph Cuts".
C++
56
star
47

VINS-Fisheye

Fisheye version of VINS-Fusion
C++
52
star
48

GeometricPretraining

This is the code base for paper ``Geometric Pretraining for Monocular Depth Estimation``, the paper is currently under review. The preprint will be available when it is ready.
49
star
49

APACE

APACE: Agile and Perception-aware Trajectory Generation for Quadrotor Flights (ICRA2024)
C++
37
star
50

plan_utils

Some useful pkgs for running planning simulation.
Makefile
29
star
51

edge_alignment

Clone of https://github.com/mpkuse/edge_alignment
C++
26
star
52

mockasimulator

C++
21
star
53

probabilistic_mapping

Probabilistic Dense Mapping
C++
19
star
54

swarm_gcs

Ground Station Software for aerial robots.
JavaScript
18
star
55

stTraj

Spatial-temporal Trajectory Planning for UAV Teach-and-Repeat
15
star
56

MASSTAR

A Multi-modal Large-scale Scene Dataset and A Versatile Toolchain for Scene Prediction
13
star
57

SLIM

7
star
58

mockacam

Camera package of mocka WFB
C++
3
star
59

RI_Mocap

Mocap device driver of RI
C++
1
star