Train in Germany, Test in The USA: Making 3D Object Detectors Generalize

This paper has been accpeted by Conference on Computer Vision and Pattern Recognition (CVPR) 2020.

Train in Germany, Test in The USA: Making 3D Object Detectors Generalize

by Yan Wang*, Xiangyu Chen*, Yurong You, Li Erran, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger, Wei-Lun Chao*

Dependencies

• Python 3.6.10
• PyTorch(1.0.0)

Usage

Prepare Datasets (Jupyter notebook)

We develop our method on these datasets:

• KITTI object detection 3D dataset
• Argoverse dataset v1.1
• nuScenes dataset v1.0
• Lyft Level 5 dataset v1.02
• Waymo dataset v1.0

1. Configure dataset_path in config_path.py.

   Raw datasets will be organized as the following structure:

    dataset_path/
        | kitti/               # KITTI object detection 3D dataset
            | training/
            | testing/
        | argo/                # Argoverse dataset v1.1
            | train1/
            | train2/
            | train3/
            | train4/
            | val/
            | test/
        | nusc/                # nuScenes dataset v1.0
            | maps/
            | samples/
            | sweeps/
            | v1.0-trainval/
        | lyft/                # Lyft Level 5 dataset v1.02
            | v1.02-train/
        | waymo/               # Waymo dataset v1.0
            | training/
            | validation/
    

2. Download all datasets.

   For KITTI, Argoverse and Waymo, we provide scripts for automatic download.

   cd scripts/
   python download.py [--datasets kitti+argo+waymo]

   nuScenes and Lyft need to downloaded manually.

3. Convert all datasets to KITTI format.

   cd scripts/
   python -m pip install -r convert_requirements.txt
   python convert.py [--datasets argo+nusc+lyft+waymo]

4. Split validation set

   We provide the train/val split used in our experiments under split folder.

   cd split/
   python replace_split.py

5. Generate car subset

   We filter scenes and only keep those with cars.

   cd scripts/
   python gen_car_split.py

Statistical Normalization (Jupyter notebook)

1. Compute car size statistics of each dataset. The computed statistics are stored as label_stats_{train/val/test}.json under KITTI format dataset root.

   cd stat_norm/
   python stat.py

2. Generate rescaled datasets according to car size statistics. The rescaled datasets are stored under $dataset_path/rescaled_datasets by default.

   cd stat_norm/
   python norm.py [--path $PATH]

Training (To be updated)

We use PointRCNN to validate our method.

1. Setup PointRCNN

   cd pointrcnn/
   ./build_and_install.sh

2. Build datasets in PointRCNN format.

   cd pointrcnn/tools/
   python generate_multi_data.py
   python generate_gt_database.py --root ...

   The NuScence dataset has much less points in each bounding box, so we have to turn of the GT_AUG_HARD_RATIO augmentation.

3. Download the models pretrained on source domains from google drive using gdrive.

   cd pointrcnn/tools/
   gdrive download -r 14MXjNImFoS2P7YprLNpSmFBsvxf5J2Kw

4. Adapt to a new domain by re-training with rescaled data.

   cd pointrcnn/tools/
   
   python train_rcnn.py --cfg_file ...

Inference

cd pointrcnn/tools/
python eval_rcnn.py --ckpt /path/to/checkpoint.pth --dataset $dataset --output_dir $output_dir 

Evaluation

We provide evaluation code with

• old (based on bbox height) and new (based on distance) difficulty metrics
• output transformation functions to locate domain gap

python evaluate/
python evaluate.py --result_path $predictions --dataset_path $dataset_root --metric [old/new]

Citation

@inproceedings{wang2020train,
  title={Train in germany, test in the usa: Making 3d object detectors generalize},
  author={Yan Wang and Xiangyu Chen and Yurong You and Li Erran and Bharath Hariharan and Mark Campbell and Kilian Q. Weinberger and Wei-Lun Chao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={11713-11723},
  year={2020}
}