• Stars
    star
    161
  • Rank 233,470 (Top 5 %)
  • Language
    Python
  • License
    GNU General Publi...
  • Created almost 4 years ago
  • Updated 12 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Person detector for 2D range data. Code release for Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera (https://arxiv.org/abs/2012.08890)

Person Detection in 2D Range Data

This repository implements DROW3 (arXiv) and DR-SPAAM (arXiv), real-time person detectors using 2D LiDARs mounted at ankle or knee height. Also included are experiments from Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera (arXiv). Pre-trained models (using PyTorch 1.6) can be found in this Google drive.

News

[06-03-2021] Our work has been accepted to ICRA'21! Checkout the presentation video here.

Quick start

First clone and install the repository

git clone https://github.com/VisualComputingInstitute/2D_lidar_person_detection.git
cd dr_spaam
python setup.py install

Use the Detector class to run inference

import numpy as np
from dr_spaam.detector import Detector

ckpt = 'path_to_checkpoint'
detector = Detector(
    ckpt,
    model="DROW3",          # Or DR-SPAAM
    gpu=True,               # Use GPU
    stride=1,               # Optionally downsample scan for faster inference
    panoramic_scan=True     # Set to True if the scan covers 360 degree
)

# tell the detector field of view of the LiDAR
laser_fov_deg = 360
detector.set_laser_fov(laser_fov_deg)

# detection
num_pts = 1091
while True:
    # create a random scan
    scan = np.random.rand(num_pts)  # (N,)

    # detect person
    dets_xy, dets_cls, instance_mask = detector(scan)  # (M, 2), (M,), (N,)

    # confidence threshold
    cls_thresh = 0.5
    cls_mask = dets_cls > cls_thresh
    dets_xy = dets_xy[cls_mask]
    dets_cls = dets_cls[cls_mask]

ROS node

We provide an example ROS node dr_spaam_ros. First install dr_spaam to your python environment. Then compile the ROS package

catkin build dr_spaam_ros

Modify the topics and the path to the pre-trained checkpoint at dr_spaam_ros/config/ and launch the node

roslaunch dr_spaam_ros dr_spaam_ros.launch

For testing, you can play a rosbag sequence from JRDB dataset. For example,

rosbag play JRDB/test_dataset/rosbags/tressider-2019-04-26_0.bag

and use RViz to visualize the inference result. A simple RViz config is located at dr_spaam_ros/example.rviz.

In addition, if you want to test with DROW dataset, you can convert a DROW sequence to a rosbag

python scripts/drow_data_converter.py --seq <PATH_TO_SEQUENCE> --output drow.bag

Training and evaluation

Download the DROW dataset and the JackRabbot dataset, and put them under dr_spaam/data as below.

dr_spaam
โ”œโ”€โ”€ data
โ”‚   โ”œโ”€โ”€ DROWv2-data
โ”‚   โ”‚   โ”œโ”€โ”€ test
โ”‚   โ”‚   โ”œโ”€โ”€ train
โ”‚   โ”‚   โ”œโ”€โ”€ val
โ”‚   โ”œโ”€โ”€ JRDB
โ”‚   โ”‚   โ”œโ”€โ”€ test_dataset
โ”‚   โ”‚   โ”œโ”€โ”€ train_dataset
...

First preprocess the JRDB dataset (extract laser measurements from raw rosbag and synchronize with images)

python bin/setup_jrdb_dataset.py

To train a network (or evaluate a pretrained checkpoint), run

python bin/train.py --cfg net_cfg.yaml [--ckpt ckpt_file.pth --evaluation]

where net_cfg.yaml specifies configuration for the training (see examples under cfgs).

Self-supervised training with a calibrated camera

If your robot has a calibrated camera (i.e. the transformation between the camera to the LiDAR is known), you can generate pseudo labels automatically during deployment and fine-tune the detector (no manual labeling needed). We provide a wrapper function dr_spaam.pseudo_labels.get_regression_target_using_bounding_boxes() for generating pseudo labels conveniently. For experiments using pseudo labels, checkout our paper Self-Supervised Person Detection in 2D Range Data using a Calibrated Camera (arXiv). Use checkpoints in this Google drive to reproduce our results.

Inference time

On DROW dataset (450 points, 225 degrees field of view)

AP0.3 AP0.5 FPS (RTX 2080 laptop) FPS (Jetson AGX Xavier)
DROW3 0.638 0.659 115.7 24.9
DR-SPAAM 0.707 0.723 99.6 22.5

On JackRabbot dataset (1091 points, 360 degrees field of view)

AP0.3 AP0.5 FPS (RTX 2080 laptop) FPS (Jetson AGX Xavier)
DROW3 0.762 0.829 35.6 10.0
DR-SPAAM 0.785 0.849 29.4 8.8

Note: Evaluation on DROW and JackRabbot are done using different models (the APs are not comparable cross dataset). Inference time was measured with PyTorch 1.7 and CUDA 10.2 on RTX 2080 laptop, and PyTorch 1.6 and L4T 4.4 on Jetson AGX Xavier.

Citation

If you use this repo in your project, please cite:

@article{Jia2021Person2DRange,
  title        = {{Self-Supervised Person Detection in 2D Range Data using a
                   Calibrated Camera}},
  author       = {Dan Jia and Mats Steinweg and Alexander Hermans and Bastian Leibe},
  booktitle    = {International Conference on Robotics and Automation (ICRA)},
  year         = {2021}
}

@inproceedings{Jia2020DRSPAAM,
  title        = {{DR-SPAAM: A Spatial-Attention and Auto-regressive
                   Model for Person Detection in 2D Range Data}},
  author       = {Dan Jia and Alexander Hermans and Bastian Leibe},
  booktitle    = {International Conference on Intelligent Robots and Systems (IROS)},
  year         = {2020}
}

More Repositories

1

triplet-reid

Code for reproducing the results of our "In Defense of the Triplet Loss for Person Re-Identification" paper.
Python
764
star
2

TrackR-CNN

TrackR-CNN baseline method for Multi-Object Tracking and Segmentation (MOTS)
Python
520
star
3

diffusion-e2e-ft

Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
Python
275
star
4

mots_tools

Tools for evaluating and visualizing results for the Multi Object Tracking and Segmentation (MOTS) task
Python
222
star
5

SiamR-CNN

Siam R-CNN two-stage re-detector for visual object tracking
Python
218
star
6

vkitti3D-dataset

Python
100
star
7

3d-semantic-segmentation

This work is based on our paper Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds, which is appeared at the IEEE International Conference on Computer Vision (ICCV) 2017, 3DRMS Workshop.
Python
98
star
8

towards-reid-tracking

Code for the paper "Towards a Principled Integration of Multi-Camera Re-Identification and Tracking through Optimal Bayes Filters"
Python
82
star
9

DR-SPAAM-Detector

DR-SPAAM: A Spatial-Attention and Auto-regressive Model for Person Detection in 2D Range Data
Python
78
star
10

ShapePriors_GCPR16

C++
47
star
11

DROW

All code related to the "DROW: Real-Time Deep Learning based Wheelchair Detection in 2D Range Data" paper
Jupyter Notebook
41
star
12

Person_MinkUNet

Person-MinkUNet. Winner of JRDB 3D detection challenge in JRDB-ACT Workshop at CVPR 2021. https://arxiv.org/abs/2107.06780
Python
20
star
13

BiternionNets-ROS

An implementation of BiternionNets for ROS, ready to run on a robot.
Python
13
star
14

mots_trackingonly_tools

Tools for Challenge 3: Tracking Only (MOT+KITTI) of MOTChallenge 2020
Python
8
star
15

Beacon8

A Torch-inspired library for high-level deep learning with Theano.
Python
5
star
16

RovinaSemanticSegmentation

Semantic segmentation code for the ROVINA project.
C++
4
star
17

CROWDBOT_perception

This is the perception pipeline for the CROWDBOT project, featuring person detection and tracking from multi-sensor modalities.
C
3
star
18

cityscapes-util

Utility toolbox for dealing with the CityScapes dataset.
Python
3
star
19

omni3d-rgbd

2
star
20

PARIS-sem-seg

A straight forward network for semantic segmentation in TensorFlow
Python
2
star
21

ROS-laserdumper

Dump ROS LaserScan data into csv files.
C++
1
star