• Stars
    star
    4,967
  • Rank 8,419 (Top 0.2 %)
  • Language
    C++
  • License
    GNU General Publi...
  • Created over 7 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Robust and Versatile Monocular Visual-Inertial State Estimator

VINS-Mono

A Robust and Versatile Monocular Visual-Inertial State Estimator

11 Jan 2019: An extension of VINS, which supports stereo cameras / stereo cameras + IMU / mono camera + IMU, is published at VINS-Fusion

29 Dec 2017: New features: Add map merge, pose graph reuse, online temporal calibration function, and support rolling shutter camera. Map reuse videos:

cla icra

VINS-Mono is a real-time SLAM framework for Monocular Visual-Inertial Systems. It uses an optimization-based sliding window formulation for providing high-accuracy visual-inertial odometry. It features efficient IMU pre-integration with bias correction, automatic estimator initialization, online extrinsic calibration, failure detection and recovery, loop detection, and global pose graph optimization, map merge, pose graph reuse, online temporal calibration, rolling shutter support. VINS-Mono is primarily designed for state estimation and feedback control of autonomous drones, but it is also capable of providing accurate localization for AR applications. This code runs on Linux, and is fully integrated with ROS. For iOS mobile implementation, please go to VINS-Mobile.

Authors: Tong Qin, Peiliang Li, Zhenfei Yang, and Shaojie Shen from the HKUST Aerial Robotics Group

Videos:

euroc indoor_outdoor AR_demo

EuRoC dataset; Indoor and outdoor performance; AR application;

MAV platform Mobile platform

MAV application; Mobile implementation (Video link for mainland China friends: Video1 Video2 Video3 Video4 Video5)

Related Papers

  • Online Temporal Calibration for Monocular Visual-Inertial Systems, Tong Qin, Shaojie Shen, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, 2018), best student paper award pdf

  • VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, Tong Qin, Peiliang Li, Zhenfei Yang, Shaojie Shen, IEEE Transactions on Roboticspdf

If you use VINS-Mono for your academic research, please cite at least one of our related papers.bib

1. Prerequisites

1.1 Ubuntu and ROS Ubuntu 16.04. ROS Kinetic. ROS Installation additional ROS pacakge

    sudo apt-get install ros-YOUR_DISTRO-cv-bridge ros-YOUR_DISTRO-tf ros-YOUR_DISTRO-message-filters ros-YOUR_DISTRO-image-transport

1.2. Ceres Solver Follow Ceres Installation, remember to make install. (Our testing environment: Ubuntu 16.04, ROS Kinetic, OpenCV 3.3.1, Eigen 3.3.3)

2. Build VINS-Mono on ROS

Clone the repository and catkin_make:

    cd ~/catkin_ws/src
    git clone https://github.com/HKUST-Aerial-Robotics/VINS-Mono.git
    cd ../
    catkin_make
    source ~/catkin_ws/devel/setup.bash

3. Visual-Inertial Odometry and Pose Graph Reuse on Public datasets

Download EuRoC MAV Dataset. Although it contains stereo cameras, we only use one camera. The system also works with ETH-asl cla dataset. We take EuRoC as the example.

3.1 visual-inertial odometry and loop closure

3.1.1 Open three terminals, launch the vins_estimator , rviz and play the bag file respectively. Take MH_01 for example

    roslaunch vins_estimator euroc.launch 
    roslaunch vins_estimator vins_rviz.launch
    rosbag play YOUR_PATH_TO_DATASET/MH_01_easy.bag 

(If you fail to open vins_rviz.launch, just open an empty rviz, then load the config file: file -> Open Config-> YOUR_VINS_FOLDER/config/vins_rviz_config.rviz)

3.1.2 (Optional) Visualize ground truth. We write a naive benchmark publisher to help you visualize the ground truth. It uses a naive strategy to align VINS with ground truth. Just for visualization. not for quantitative comparison on academic publications.

    roslaunch benchmark_publisher publish.launch  sequence_name:=MH_05_difficult

(Green line is VINS result, red line is ground truth).

3.1.3 (Optional) You can even run EuRoC without extrinsic parameters between camera and IMU. We will calibrate them online. Replace the first command with:

    roslaunch vins_estimator euroc_no_extrinsic_param.launch

No extrinsic parameters in that config file. Waiting a few seconds for initial calibration. Sometimes you cannot feel any difference as the calibration is done quickly.

3.2 map merge

After playing MH_01 bag, you can continue playing MH_02 bag, MH_03 bag ... The system will merge them according to the loop closure.

3.3 map reuse

3.3.1 map save

Set the pose_graph_save_path in the config file (YOUR_VINS_FOLEDER/config/euroc/euroc_config.yaml). After playing MH_01 bag, input s in vins_estimator terminal, then enter. The current pose graph will be saved.

3.3.2 map load

Set the load_previous_pose_graph to 1 before doing 3.1.1. The system will load previous pose graph from pose_graph_save_path. Then you can play MH_02 bag. New sequence will be aligned to the previous pose graph.

4. AR Demo

4.1 Download the bag file, which is collected from HKUST Robotic Institute. For friends in mainland China, download from bag file.

4.2 Open three terminals, launch the ar_demo, rviz and play the bag file respectively.

    roslaunch ar_demo 3dm_bag.launch
    roslaunch ar_demo ar_rviz.launch
    rosbag play YOUR_PATH_TO_DATASET/ar_box.bag 

We put one 0.8m x 0.8m x 0.8m virtual box in front of your view.

5. Run with your device

Suppose you are familiar with ROS and you can get a camera and an IMU with raw metric measurements in ROS topic, you can follow these steps to set up your device. For beginners, we highly recommend you to first try out VINS-Mobile if you have iOS devices since you don't need to set up anything.

5.1 Change to your topic name in the config file. The image should exceed 20Hz and IMU should exceed 100Hz. Both image and IMU should have the accurate time stamp. IMU should contain absolute acceleration values including gravity.

5.2 Camera calibration:

We support the pinhole model and the MEI model. You can calibrate your camera with any tools you like. Just write the parameters in the config file in the right format. If you use rolling shutter camera, please carefully calibrate your camera, making sure the reprojection error is less than 0.5 pixel.

5.3 Camera-Imu extrinsic parameters:

If you have seen the config files for EuRoC and AR demos, you can find that we can estimate and refine them online. If you familiar with transformation, you can figure out the rotation and position by your eyes or via hand measurements. Then write these values into config as the initial guess. Our estimator will refine extrinsic parameters online. If you don't know anything about the camera-IMU transformation, just ignore the extrinsic parameters and set the estimate_extrinsic to 2, and rotate your device set at the beginning for a few seconds. When the system works successfully, we will save the calibration result. you can use these result as initial values for next time. An example of how to set the extrinsic parameters is inextrinsic_parameter_example

5.4 Temporal calibration: Most self-made visual-inertial sensor sets are unsynchronized. You can set estimate_td to 1 to online estimate the time offset between your camera and IMU.

5.5 Rolling shutter: For rolling shutter camera (carefully calibrated, reprojection error under 0.5 pixel), set rolling_shutter to 1. Also, you should set rolling shutter readout time rolling_shutter_tr, which is from sensor datasheet(usually 0-0.05s, not exposure time). Don't try web camera, the web camera is so awful.

5.6 Other parameter settings: Details are included in the config file.

5.7 Performance on different devices:

(global shutter camera + synchronized high-end IMU, e.g. VI-Sensor) > (global shutter camera + synchronized low-end IMU) > (global camera + unsync high frequency IMU) > (global camera + unsync low frequency IMU) > (rolling camera + unsync low frequency IMU).

6. Docker Support

To further facilitate the building process, we add docker in our code. Docker environment is like a sandbox, thus makes our code environment-independent. To run with docker, first make sure ros and docker are installed on your machine. Then add your account to docker group by sudo usermod -aG docker $YOUR_USER_NAME. Relaunch the terminal or logout and re-login if you get Permission denied error, type:

cd ~/catkin_ws/src/VINS-Mono/docker
make build
./run.sh LAUNCH_FILE_NAME   # ./run.sh euroc.launch

Note that the docker building process may take a while depends on your network and machine. After VINS-Mono successfully started, open another terminal and play your bag file, then you should be able to see the result. If you need modify the code, simply run ./run.sh LAUNCH_FILE_NAME after your changes.

7. Acknowledgements

We use ceres solver for non-linear optimization and DBoW2 for loop detection, and a generic camera model.

8. Licence

The source code is released under GPLv3 license.

We are still working on improving the code reliability. For any technical issues, please contact Tong QIN <tong.qinATconnect.ust.hk> or Peiliang LI <pliapATconnect.ust.hk>.

For commercial inquiries, please contact Shaojie SHEN <eeshaojieATust.hk>

More Repositories

1

VINS-Fusion

An optimization-based multi-sensor state estimator
C++
3,181
star
2

Fast-Planner

A Robust and Efficient Trajectory Planner for Quadrotors
C++
2,433
star
3

A-LOAM

Advanced implementation of LOAM
C++
1,957
star
4

VINS-Mobile

Monocular Visual-Inertial State Estimator on Mobile Phones
C++
1,269
star
5

Teach-Repeat-Replan

Teach-Repeat-Replan: A Complete and Robust System for Aggressive Flight in Complex Environments
C++
951
star
6

GVINS

Tightly coupled GNSS-Visual-Inertial system for locally smooth and globally consistent state estimation in complex environment.
C++
882
star
7

FUEL

An Efficient Framework for Fast UAV Exploration
C++
744
star
8

Stereo-RCNN

Code for 'Stereo R-CNN based 3D Object Detection for Autonomous Driving' (CVPR 2019)
Python
690
star
9

DenseSurfelMapping

This is the open-source version of ICRA 2019 submission "Real-time Scalable Dense Surfel Mapping"
C++
661
star
10

FIESTA

Fast Incremental Euclidean Distance Fields for Online Motion Planning of Aerial Robots
C++
617
star
11

EPSILON

C++
493
star
12

ESVO

This repository maintains the implementation of "Event-based Stereo Visual Odometry".
C++
408
star
13

Btraj

Bezier Trajectory Generation for Autonomous Quadrotor, ICRA 2018
C++
407
star
14

grad_traj_optimization

Gradient-Based Online Safe Trajectory Generator
C++
363
star
15

MonoLaneMapping

Online Monocular Lane Mapping Using Catmull-Rom Spline (IROS 2023)
Python
349
star
16

open_quadtree_mapping

This is a monocular dense mapping system corresponding to IROS 2018 "Quadtree-accelerated Real-time Monocular Dense Mapping"
Cuda
347
star
17

MVDepthNet

This repository provides PyTorch implementation for 3DV 2018 paper "MVDepthNet: real-time multiview depth estimation neural network"
Python
305
star
18

D2SLAM

$D^2$SLAM: Decentralized and Distributed Collaborative Visual-inertial SLAM System for Aerial Swarm
Jupyter Notebook
277
star
19

OmniNxt

[IROS 2024 Oral] A Fully Open-source and Compact Aerial Robot with Omnidirectional Visual Perception
255
star
20

G3Reg

A fast and robust global registration library for outdoor LiDAR point clouds.
C++
200
star
21

GVINS-Dataset

A dataset containing synchronized visual, inertial and GNSS raw measurements.
C++
197
star
22

Nxt-FC

Mini PX4 for UAV Group
Shell
187
star
23

Omni-swarm

A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swar.
Jupyter Notebook
179
star
24

spatiotemporal_semantic_corridor

Implementation of the paper "Safe Trajectory Generation For Complex Urban Environments Using Spatio-temporal Semantic Corridor".
C++
160
star
25

PredRecon

[ICRA 2023] A Prediction-boosted Planner for Fast and High-quality Autonomous Aerial Reconstruction
C++
156
star
26

FC-Planner

[ICRA 2024 Best UAV Paper Award Finalist] An Efficient Gloabl Planner for Aerial Coverage
C++
155
star
27

eudm_planner

Implementation of the paper "Efficient Uncertainty-aware Decision-making for Automated Driving Using Guided Branching".
C++
139
star
28

mockamap

a simple map generator based on ROS
C++
133
star
29

DSP

Trajectory Prediction with Graph-based Dual-scale Context Fusion
Python
132
star
30

pointcloudTraj

Trajectory generation on point clouds
C++
128
star
31

Pagor

Pyramid Semantic Graph-based Global Point Cloud Registration with Low Overlap (IROS 2023)
C++
127
star
32

Flow-Motion-Depth

This is the project page of the paper "Flow-Motion and Depth Network for Monocular Stereo and Beyond''
Python
114
star
33

gnss_comm

Basic definitions and utility functions for GNSS raw measurement processing
C++
111
star
34

SIMPL

SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving
Python
107
star
35

VINS-kidnap

a place recognition system for VINS-fusion
105
star
36

ublox_driver

A driver for u-blox receiver (ZED-F9P) with ros support
C++
102
star
37

TopoTraj

A robust UAV local planner based on the ICRA2020 paper: Robust Real-time UAV Replanning Using Guided Gradient-based Optimization and Topological Paths
90
star
38

TimeOptimizer

Optimal Time Allocation for Quadrotor Trajectory Generation
C++
83
star
39

AutoTrans

AutoTrans: A Complete Planning and Control Framework for Autonomous UAV Payload Transportation.
C++
76
star
40

LiDAR-Registration-Benchmark

LiDAR-based 3D global registration benchmark.
Python
75
star
41

Pinhole-Fisheye-Mapping

70
star
42

UniQuad

UniQuad: A Unified and Versatile Quadrotor Platform Series for UAV Research and Application
67
star
43

IMPACTOR

Impact-Aware Planning and Control for Aerial Robots with Suspended Payloads
C
67
star
44

SLABIM

An open-sourced SLAM dataset that couples with BIM (Building Information Modeling).
Python
66
star
45

HKUST-ELEC5660-Introduction-to-Aerial-Robotics

Repo for HKUST ELEC5660 Course Notes & Lab Tutorial & Project Docker
C++
57
star
46

EMSGC

This repository maintains the implementation of the paper "Event-based Motion Segmentation withSpatio-Temporal Graph Cuts".
C++
56
star
47

VINS-Fisheye

Fisheye version of VINS-Fusion
C++
52
star
48

GeometricPretraining

This is the code base for paper ``Geometric Pretraining for Monocular Depth Estimation``, the paper is currently under review. The preprint will be available when it is ready.
49
star
49

APACE

APACE: Agile and Perception-aware Trajectory Generation for Quadrotor Flights (ICRA2024)
C++
37
star
50

plan_utils

Some useful pkgs for running planning simulation.
Makefile
29
star
51

edge_alignment

Clone of https://github.com/mpkuse/edge_alignment
C++
26
star
52

mockasimulator

C++
21
star
53

probabilistic_mapping

Probabilistic Dense Mapping
C++
19
star
54

swarm_gcs

Ground Station Software for aerial robots.
JavaScript
18
star
55

stTraj

Spatial-temporal Trajectory Planning for UAV Teach-and-Repeat
15
star
56

MASSTAR

A Multi-modal Large-scale Scene Dataset and A Versatile Toolchain for Scene Prediction
13
star
57

SLIM

7
star
58

mockacam

Camera package of mocka WFB
C++
3
star
59

RI_Mocap

Mocap device driver of RI
C++
1
star