• Stars
    star
    105
  • Rank 328,196 (Top 7 %)
  • Language
  • Created over 5 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

a place recognition system for VINS-fusion

VINS-Fusion with Cerebro

This is the cerebro module for VINS-Fusion. The aim of this project is better loop detection and recover from kidnap. The cerebro node connects to the vins_estimator nodes of VINS-Fusion (with ros interface). The DataManager class handles all the incoming data. Visualization handles all the visualization. Cerebro class handles the loop closure intelligence part. It publishes a LoopMsg which contains timestamps of the identified loopcandidate along with the computed relative pose between the pair. The pose computation needs a stereo pair for reliable pose computation. This is a multi-threaded object oriented implementation and I observe a CPU load factor of about 2.0. A separate node handles pose graph solver (it is in github-repo ).

A MORE UPTO DATE README IS AVAILABLE IN THE ORIGINAL REPO (mpkuse/cerebro )

Manuscript: Preprint : https://arxiv.org/abs/1904.06962

Highlight Video

IMAGE ALT TEXT

Alternate link to highlight video: Dailymotion

AR Demo Under Kidnap.

We show our system's performance for AR tasks under kidnap. We are able to track and relocalize despite long kidnaps. IMAGE ALT TEXT

Save Map to Disk and Relocalize from Loaded Map (Teach-Repeat)

Our system is able to save the constructed map to disk. It is also able to load a previously stored map and relocalize live and in realtime from that map. We demo the effect of relocalization from map with our group's SurfelMapping code

IMAGE ALT TEXT


MyntEye Demo (Using VINS-Fusion as Odometry Estimator)

IMAGE ALT TEXT

IMAGE ALT TEXT

MyntEye Demo (Using VINS-Mono as Odometry Estimator)

IMAGE ALT TEXT

IMAGE ALT TEXT

EuRoC MAV Dataset live merge MH-01, ... MH-05.

IMAGE ALT TEXT

EuRoC MAV Dataset live merge V1_01, V1_02, V1_03, V2_01, V2_02

IMAGE ALT TEXT

For more demonstration, have a look at my youtube playlist

Visual-Inertial Datasets

How to run - Docker

I highly recommend the already deployed packages with docker. Run the roscore on your host pc and all the packages run inside of docker container. rviz runs on the host pc.

I assume you have a PC with a graphics card and cuda9 working smoothly and nvidia-docker installed.

$(host) export ROS_HOSTNAME=`hostname`
$(host) roscore
# assume that host has the ip address 172.17.0.1 in docker-network aka docker0
$(host) docker run --runtime=nvidia -it \
        --add-host `hostname`:172.17.0.1 \
        --env ROS_MASTER_URI=http://`hostname`:11311/ \
        --env CUDA_VISIBLE_DEVICES=0 \
        --hostname happy_go \
        --name happy_go  \
        mpkuse/kusevisionkit:vins-kidnap bash
$(host) rviz # inside rviz open config cerebro/config/good-viz.rviz. If you open rviz in a new tab you might need to do set ROS_HOSTNAME again.
$(docker) roslaunch cerebro mynteye_vinsfusion.launch
            OR
$(docker) roslaunch cerebro euroc_vinsfusion.launch
$(host) rosbag play 1.bag

Edit the launch file as needed.

If you are unfamiliar with docker, you may want to read my blog post on using docker for computer vision researchers. You might want to have a look at my test ros-package to ensure things work with docker docker_ros_test.

How to compile (from scratch)

You will need a) VINS-Fusion (with modification for reset by mpkuse), b) cerebro and c) solve_keyframe_pose_graph. Besure to setup a catkin_ws and make sure your ROS works correctly.

Dependencies

  • ROS Kinetic
  • Eigen3
  • Ceres
  • OpenCV3 (should also work with 2.4 (not tested), 3.3 and 3.4)
  • Theia-sfm - OpenImageIO. tested with version Release-1.7.6RC1 - RocksDB. tested with version v5.9.2

Get VINS-Fusion Working GIT

I recommend you use my fork of VINS-Fusion, in which I have fixed some bugs and added mechanism for reseting the VINS.

cd catkin_ws/src
#git clone https://github.com/HKUST-Aerial-Robotics/VINS-Fusion.git
git clone https://github.com/mpkuse/VINS-Fusion
cd ../
catkin_make
source catkin_ws/devel/setup.bash

Make sure your vins-fusion can compile and run correctly. See vins-fusion github repo for the latest information on prerequisites and compilation instructions. For compatibility I recommend using my fork of vins-mono/vins-fusion. Some minor modifications have been made by me for working with kidnap cases.

Cerebro GIT

cd catkin_ws/src/
git clone https://github.com/mpkuse/cerebro
cd ../
catkin_make

This has 2 exectables. a) ros server that takes as input an image and returns a image descriptor. b) cerebro_node, this finds the loop candidates and computes the relative poses. I have also included my trained model (about 4 MB) in this package (located scripts/keras.model). The pose computation uses the stereo pair in this node. This node publishes the loopcandidate's relative pose which is expected to be consumed the pose-graph solver.

If you wish to train your own model, you may use my learning code here.

Threads:

  • Main Thread : ros-callbacks
  • data_association_th : Sync the incoming image data and incoming data vins_estimator.
  • desc_th : Consumes the images to produce whole-image-descriptors.
  • dot_product_th : Dot product of current image descriptor with all the previous ones.
  • loopcandidate_consumer_th : Computes the relative pose at the loopcandidates. Publishes the loopEdge.
  • kidnap_th : Identifies kidnap. If kidnap publishes the reset signals for vins_estimator.
  • viz_th : Publishes the image-pair, and more things for debugging and analysis.

Pose Graph Solver GIT

Use my pose graph solver, github-repo. The differences between this implementation and the original from VINS-Fusion is that this can handle kidnap cases, handles multiple world co-ordinate frames and it uses a switch-constraint formulation of the pose-graph problem. It uses the disjoint set forest to maintain a set association of world co-ordinate systems.

cd catkin_ws/src/
git clone https://github.com/mpkuse/solve_keyframe_pose_graph
cd ../
catkin_make

Threads:

  • Main thread : ros-callbacks for odometry poses (from vins_estimator) and LoopMsg (from cerebro).
  • SLAM : Monitors the node-poses and loop-edges, on new loop-edges constructs and solves the pose-graph optimization problem.
  • th4 : Publish latest odometry.
  • th5 : Display image to visualize disjoint set datastructure.
  • th6 : Publish corrected poses, Different color for nodes in different co-ordinate systems.

AR Demo GIT

To make ar_demo similar to my video above you could use this package. It takes the corrected pose from the pose graph solver (along with the worldID) and renders a polygonal mesh on the camera-image. It also have support for ground plane estimation, checkout the readme of that package.

cd catkin_ws/src
git clone https://github.com/mpkuse/ar_demo
cd ../
catkin_make

vins_mono_debug_pkg (optional, needed only if you wish to debug vins-mono/vins-fusion)

With cerebro node it is possible to live run the vins and make it log all the details to file for further analysis/debugging. This might be useful for researchers and other Ph.D. student to help VINS-Fusion improve further. see github/mpkuse/vins_mono. It basically contains unit tests and some standalone tools which might come in handy. If you are looking to help improve VINS-fusion or cerebro also look at 'Development Guidelines'.

How to run the Full System

roslaunch cerebro mynteye_vinsfusion.launch

You can get some of my bag files collected with the mynteye camera HERE. More example launch files in folder launch, all the config files which contains calibration info are in folder config.

Development Guidelines

If you are developing I still recommend using docker. with -v flags in docker you could mount your pc's folders on the docker. I recommend keeping all the packages in folder docker_ws_slam/catkin_ws/src on your host pc. And all the rosbags in folder /media/mpkuse/Bulk_Data. And then mount these two folders on the docker-container. Edit the following command as needed.

docker run --runtime=nvidia -it  -v /media/mpkuse/Bulk_Data/:/Bulk_Data  -v /home/mpkuse/docker_ws_slam:/app  --add-host `hostname`:172.17.0.1  --env ROS_MASTER_URI=http://`hostname`:11311/  --env CUDA_VISIBLE_DEVICES=0  --hostname happy_go   --name happy_go  mpkuse/kusevisionkit:ros-kinetic-vins bash

Each of my classes can export the data they hold as json objects and image files. Look at the end of main() in cerebro_node.cpp and modify as needed to extract more debug data. Similarly the pose graph solver can also be debugged. For streamlining printing messages I have preprocessor macros at the start of function implementation (of classes DataManager and Cerebro), read the comments there and edit as per need. Try to implement your algorithms in object oriented way and using the producer-consumer paradigm. Look at my thread-mains for example.

Finally, sensible PR with bug fixes, enhancements are welcome!

Authors

Manohar Kuse [email protected]

More Repositories

1

VINS-Mono

A Robust and Versatile Monocular Visual-Inertial State Estimator
C++
4,967
star
2

VINS-Fusion

An optimization-based multi-sensor state estimator
C++
3,181
star
3

Fast-Planner

A Robust and Efficient Trajectory Planner for Quadrotors
C++
2,433
star
4

A-LOAM

Advanced implementation of LOAM
C++
1,957
star
5

VINS-Mobile

Monocular Visual-Inertial State Estimator on Mobile Phones
C++
1,269
star
6

Teach-Repeat-Replan

Teach-Repeat-Replan: A Complete and Robust System for Aggressive Flight in Complex Environments
C++
951
star
7

GVINS

Tightly coupled GNSS-Visual-Inertial system for locally smooth and globally consistent state estimation in complex environment.
C++
882
star
8

FUEL

An Efficient Framework for Fast UAV Exploration
C++
744
star
9

Stereo-RCNN

Code for 'Stereo R-CNN based 3D Object Detection for Autonomous Driving' (CVPR 2019)
Python
690
star
10

DenseSurfelMapping

This is the open-source version of ICRA 2019 submission "Real-time Scalable Dense Surfel Mapping"
C++
661
star
11

FIESTA

Fast Incremental Euclidean Distance Fields for Online Motion Planning of Aerial Robots
C++
617
star
12

EPSILON

C++
493
star
13

ESVO

This repository maintains the implementation of "Event-based Stereo Visual Odometry".
C++
408
star
14

Btraj

Bezier Trajectory Generation for Autonomous Quadrotor, ICRA 2018
C++
407
star
15

grad_traj_optimization

Gradient-Based Online Safe Trajectory Generator
C++
363
star
16

MonoLaneMapping

Online Monocular Lane Mapping Using Catmull-Rom Spline (IROS 2023)
Python
349
star
17

open_quadtree_mapping

This is a monocular dense mapping system corresponding to IROS 2018 "Quadtree-accelerated Real-time Monocular Dense Mapping"
Cuda
347
star
18

MVDepthNet

This repository provides PyTorch implementation for 3DV 2018 paper "MVDepthNet: real-time multiview depth estimation neural network"
Python
305
star
19

D2SLAM

$D^2$SLAM: Decentralized and Distributed Collaborative Visual-inertial SLAM System for Aerial Swarm
Jupyter Notebook
277
star
20

OmniNxt

[IROS 2024 Oral] A Fully Open-source and Compact Aerial Robot with Omnidirectional Visual Perception
255
star
21

G3Reg

A fast and robust global registration library for outdoor LiDAR point clouds.
C++
200
star
22

GVINS-Dataset

A dataset containing synchronized visual, inertial and GNSS raw measurements.
C++
197
star
23

Nxt-FC

Mini PX4 for UAV Group
Shell
187
star
24

Omni-swarm

A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swar.
Jupyter Notebook
179
star
25

spatiotemporal_semantic_corridor

Implementation of the paper "Safe Trajectory Generation For Complex Urban Environments Using Spatio-temporal Semantic Corridor".
C++
160
star
26

PredRecon

[ICRA 2023] A Prediction-boosted Planner for Fast and High-quality Autonomous Aerial Reconstruction
C++
156
star
27

FC-Planner

[ICRA 2024 Best UAV Paper Award Finalist] An Efficient Gloabl Planner for Aerial Coverage
C++
155
star
28

eudm_planner

Implementation of the paper "Efficient Uncertainty-aware Decision-making for Automated Driving Using Guided Branching".
C++
139
star
29

mockamap

a simple map generator based on ROS
C++
133
star
30

DSP

Trajectory Prediction with Graph-based Dual-scale Context Fusion
Python
132
star
31

pointcloudTraj

Trajectory generation on point clouds
C++
128
star
32

Pagor

Pyramid Semantic Graph-based Global Point Cloud Registration with Low Overlap (IROS 2023)
C++
127
star
33

Flow-Motion-Depth

This is the project page of the paper "Flow-Motion and Depth Network for Monocular Stereo and Beyond''
Python
114
star
34

gnss_comm

Basic definitions and utility functions for GNSS raw measurement processing
C++
111
star
35

SIMPL

SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving
Python
107
star
36

ublox_driver

A driver for u-blox receiver (ZED-F9P) with ros support
C++
102
star
37

TopoTraj

A robust UAV local planner based on the ICRA2020 paper: Robust Real-time UAV Replanning Using Guided Gradient-based Optimization and Topological Paths
90
star
38

TimeOptimizer

Optimal Time Allocation for Quadrotor Trajectory Generation
C++
83
star
39

AutoTrans

AutoTrans: A Complete Planning and Control Framework for Autonomous UAV Payload Transportation.
C++
76
star
40

LiDAR-Registration-Benchmark

LiDAR-based 3D global registration benchmark.
Python
75
star
41

Pinhole-Fisheye-Mapping

70
star
42

UniQuad

UniQuad: A Unified and Versatile Quadrotor Platform Series for UAV Research and Application
67
star
43

IMPACTOR

Impact-Aware Planning and Control for Aerial Robots with Suspended Payloads
C
67
star
44

SLABIM

An open-sourced SLAM dataset that couples with BIM (Building Information Modeling).
Python
66
star
45

HKUST-ELEC5660-Introduction-to-Aerial-Robotics

Repo for HKUST ELEC5660 Course Notes & Lab Tutorial & Project Docker
C++
57
star
46

EMSGC

This repository maintains the implementation of the paper "Event-based Motion Segmentation withSpatio-Temporal Graph Cuts".
C++
56
star
47

VINS-Fisheye

Fisheye version of VINS-Fusion
C++
52
star
48

GeometricPretraining

This is the code base for paper ``Geometric Pretraining for Monocular Depth Estimation``, the paper is currently under review. The preprint will be available when it is ready.
49
star
49

APACE

APACE: Agile and Perception-aware Trajectory Generation for Quadrotor Flights (ICRA2024)
C++
37
star
50

plan_utils

Some useful pkgs for running planning simulation.
Makefile
29
star
51

edge_alignment

Clone of https://github.com/mpkuse/edge_alignment
C++
26
star
52

mockasimulator

C++
21
star
53

probabilistic_mapping

Probabilistic Dense Mapping
C++
19
star
54

swarm_gcs

Ground Station Software for aerial robots.
JavaScript
18
star
55

stTraj

Spatial-temporal Trajectory Planning for UAV Teach-and-Repeat
15
star
56

MASSTAR

A Multi-modal Large-scale Scene Dataset and A Versatile Toolchain for Scene Prediction
13
star
57

SLIM

7
star
58

mockacam

Camera package of mocka WFB
C++
3
star
59

RI_Mocap

Mocap device driver of RI
C++
1
star