FlowNet3D: Learning Scene Flow in 3D Point Clouds
Created by Xingyu Liu, Charles R. Qi and Leonidas J. Guibas from Stanford University and Facebook AI Research (FAIR).
Citation
If you find our work useful in your research, please cite:
@article{liu:2019:flownet3d,
title={FlowNet3D: Learning Scene Flow in 3D Point Clouds},
author={Liu, Xingyu and Qi, Charles R and Guibas, Leonidas J},
journal={CVPR},
year={2019}
}
Abstract
Many applications in robotics and human-computer interaction can benefit from understanding 3D motion of points in a dynamic environment, widely noted as scene flow. While most previous methods focus on stereo and RGB-D images as input, few try to estimate scene flow directly from point clouds. In this work, we propose a novel deep neural network named FlowNet3D that learns scene flow from point clouds in an end-to-end fashion. Our network simultaneously learns deep hierarchical features of point clouds and flow embeddings that represent point motions, supported by two newly proposed learning layers for point sets. We evaluate the network on both challenging synthetic data from FlyingThings3D and real Lidar scans from KITTI. Trained on synthetic data only, our network successfully generalizes to real scans, outperforming various baselines and showing competitive results to the prior art. We also demonstrate two applications of our scene flow output (scan registration and motion segmentation) to show its potential wide use cases.
Installation
Install TensorFlow. The code is tested under TF1.9.0 GPU version, g++ 5.4.0, CUDA 9.0 and Python 3.5 on Ubuntu 16.04. There are also some dependencies for a few Python libraries for data processing and visualizations like cv2. It's highly recommended that you have access to GPUs.
Compile Customized TF Operators
The TF operators are included under tf_ops, you need to compile them first by make under each ops subfolder (check Makefile). Update arch in the Makefiles for different CUDA Compute Capability that suits your GPU if necessary.
Usage
Flyingthings3d Data preparation
The data preprocessing scripts are included in data_preprocessing. To process the raw data, first download FlyingThings3D dataset. flyingthings3d__disparity.tar.bz2, flyingthings3d__disparity_change.tar.bz2, flyingthings3d__optical_flow.tar.bz2 and flyingthings3d__frames_finalpass.tar are needed. Then extract the files in /path/to/flyingthings3d such that the directory looks like
/path/to/flyingthings3d
disparity/
disparity_change/
optical_flow/
frames_finalpass/
Then cd into directory data_preprocessing and execute command to generate .npz files of processed data
python proc_dataset_gen_point_pairs_color.py --input_dir /path/to/flyingthings3d --output_dir data_processed_maxcut_35_20k_2k_8192
The processed data is also provided here for download (total size ~11GB).
Training and Evaluation
To train the model, simply execute the shell script command_train.sh. Batch size, learning rate etc are adjustable. The model used for training is model_concat_upsa.py.
sh command_train.sh
To evaluate the model, simply execute the shell script command_evaluate_flyingthings.sh.
sh command_evaluate_flyingthings.sh
A pre-trained model is provided here for download.
KITTI Experiment
We release the processed KITTI scene flow dataset here for download (total size ~266MB). The KITTI scene flow dataset was processed by converting the 2D optical flow into 3D scene flow and removing the ground points. We processed the first 150 data points from KITTI scene flow dataset. Each of the data points are stored as a .npz file and its dictionary has three keys: pos1, pos2 and gt, representing the first frame of point cloud, second frame of point cloud and the ground truth scene flow vectors for the points in the first frame.
To evaluate the FlyingThings3D trained model on KITTI without finetuning, first download the processed KITTI data and extract it into kitti_rm_ground/ directory. Then execute the shell script command_evaluate_kitti.sh.
sh command_evaluate_kitti.sh
Note that the model used for evaluation is in model_concat_upsa_eval_kitti.py instead of the model used for training.
License
Our code is released under MIT License (see LICENSE file for details).
Related Projects
- MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences by Liu et al. (ICCV 2019 Oral Presentation). Code and data released in GitHub.
- Learning Video Representations from Correspondence Proposals by Liu et al. (CVPR 2019 Oral Presentation). Code and data released in GitHub.
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation by Qi et al. (CVPR 2017 Oral Presentation). Code and data released in GitHub.
- PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space by Qi et al. (NIPS 2017). Code and data released in GitHub.