FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

This repo implements the paper FUTR3D: A Unified Sensor Fusion Framework for 3D Detection. Paper - project page

We built our implementation upon MMdetection3D 1.0.0rc6. The major part of the code is in the directory plugin/futr3d.

Environment

Prerequisite

1. mmcv-full>=1.5.2, <=1.7.0
2. mmdet>=2.24.0, <=3.0.0
3. mmseg>=0.20.0, <=1.0.0
4. nuscenes-devkit

Installation

There is no neccesary to install mmdet3d separately, please install based on this repo:

cd futr3d
pip3 install -v -e .

Data

Please follow the mmdet3d to process the data. https://mmdetection3d.readthedocs.io/en/stable/datasets/nuscenes_det.html

Notably, we have modified the nuscenes_converter.py to add the radar infomation, so the infos.pkl generated by our code is different from the original code. The other infomation except the radar infos is the same with the original infos.pkl.

Train

For example, to train FUTR3D with LiDAR only on 8 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/lidar_only/lidar_0075_900q.py 8

For LiDAR-Cam and Cam-Radar version, we need pre-trained model.

The Cam-Radar uses DETR3D model as pre-trained model, please check DETR3D.

The LiDAR-Cam uses fused LiDAR-only and Cam-only model as pre-trained model. You can use

python tools/fuse_model.py --img <cam checkpoint path> --lidar <lidar checkpoint path> --out <out model path>

to fuse cam-only and lidar-only models.

Evaluate

For example, to evalaute FUTR3D with LiDAR-cam on 8 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/lidar_cam/lidar_0075_cam_res101.py ../lidar_cam.pth 8 --eval bbox

Results

LiDAR & Cam

Cam & Radar

LiDAR only

Cam only

The camera-only version of FUTR3D is the same as DETR3D. Please check DETR3D for detail implementation.

Acknowledgment

For the implementation, we rely heavily on MMCV, MMDetection, MMDetection3D, and DETR3D

Related projects

1. DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
2. MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
3. For more projects on Autonomous Driving, check out our Visual-Centric Autonomous Driving (VCAD) project page webpage

Reference

@article{chen2022futr3d,
  title={FUTR3D: A Unified Sensor Fusion Framework for 3D Detection},
  author={Chen, Xuanyao and Zhang, Tianyuan and Wang, Yue and Wang, Yilun and Zhao, Hang},
  journal={arXiv preprint arXiv:2203.10642},
  year={2022}
}

Contact: Xuanyao Chen at: [email protected] or [email protected]