3DTrans: Autonomous Driving Transfer Learning Codebase

3DTrans is a lightweight, simple, self-contained open-source codebase for exploring the Autonomous Driving-oriented Transfer Learning Techniques, which mainly consists of four functions at present:

• Unsupervised Domain Adaptation (UDA) for 3D Point Clouds
• Active Domain Adaptation (ADA) for 3D Point Clouds
• Semi-Supervised Domain Adaptation (SSDA) for 3D Point Clouds
• Multi-dateset Domain Fusion (MDF) for 3D Point Clouds

This project is developed and maintained by Autonomous Driving Group [at] Shanghai AI Laboratory (ADLab).

Overview

• Update News
• TODO List
• 3DTrans Framework Introduction
  • What does 3DTrans Codebase do?
• Installation for 3DTrans
• Getting Started
• Model Zoo:
  • UDA Results:
  • ADA Results:
  • SSDA Results:
  • MDF Results:
• Point Cloud Pre-training for Autonomous Driving Task
• Visualization Tools for 3DTrans
• Acknowledge
• Citation

🔥 News 🔥

• Based on 3DTrans, we achieved significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint, PV-RCNN (updated on Jun. 2023).
• Two papers developed using our 3DTrans were accepted by CVPR-2023: Uni3D and Bi3D. (updated on Mar. 2023).
• Our 3DTrans supported the Semi-Supervised Domain Adaptation (SSDA) for 3D Object Detection (updated on Nov. 2022).
• Our 3DTrans supported the Active Domain Adaptation (ADA) of 3D Object Detection for achieving a good trade-off between high performance and annotation cost (updated on Oct. 2022).
• Our 3DTrans supported several typical transfer learning techniques (such as TQS, CLUE, SN, ST3D, Pseudo-labeling, SESS, and Mean-Teacher) for autonomous driving-related model adaptation and transfer.
• Our 3DTrans supported the Multi-domain Dataset Fusion (MDF) of 3D Object Detection for enabling the existing 3D models to effectively learn from multiple off-the-shelf 3D datasets (updated on Sep. 2022).
• Our 3DTrans supported the Unsupervised Domain Adaptation (UDA) of 3D Object Detection for deploying a well-trained source model to an unlabeled target domain (updated on July 2022).
• The self-learning for unlabeled point clouds and new data augmentation operations (such as Object Scaling (ROS) and Size Normalization (SN)) has been supported for 3DTrans, which can achieve a performance boost based on the dataset prior of object-size statistics

🚀 We are actively updating this repository currently, and more cross-dataset fusion solutions (including domain attention and mixture-of-experts) and more low-cost data sampling strategy will be supported by 3DTrans in the furture, which aims to boost the generalization ability and adaptability of the existing state-of-the-art models. 🚀

We expect this repository will inspire the research of 3D model generalization since it will push the limits of perceptual performance. 🗼

💪 TODO List 💪

• For ADA module, need to add the sequence-level data selection policy (to meet the requirement of practical annotation process).
• Provide experimental findings for the AD-related 3D pre-training (Our ongoing research, which currently achieves promising pre-training results towards downstream tasks by exploiting large-scale unlabeled data in ONCE dataset using 3DTrans).

3DTrans Framework Introduction

What does 3DTrans Codebase do?

• Rapid Target-domain Adaptation: 3DTrans can boost the 3D perception model's adaptability for an unseen domain only using unlabeled data. For example, ST3D supported by 3DTrans repository achieves a new state-of-the-art model transfer performance for many adaptation tasks, further boosting the transferability of detection models. Besides, 3DTrans develops many new UDA techniques to solve different types of domain shifts (such as LiDAR-induced shifts or object-size-induced shifts), which includes Pre-SN, Post-SN, and range-map downsampling retraining.

• Annotation-saving Target-domain Transfer: 3DTrans can select the most informative subset of an unseen domain and label them at a minimum cost. For example, 3DTrans develops the Bi3D, which selects partial-yet-important target data and labels them at a minimum cost, to achieve a good trade-off between high performance and low annotation cost. Besides, 3DTrans has integrated several typical transfer learning techniques into the 3D object detection pipeline. For example, we integrate the TQS, CLUE, SN, ST3D, Pseudo-labeling, SESS, and Mean-Teacher for supporting autonomous driving-related model transfer.

• Joint Training on Multiple 3D Datasets: 3DTrans can perform the multi-dataset 3D object detection task. For example, 3DTrans develops the Uni3D for multi-dataset 3D object detection, which enables the current 3D baseline models to effectively learn from multiple off-the-shelf 3D datasets, boosting the reusability of 3D data from different autonomous driving manufacturers.

• Multi-dataset Support: 3DTrans provides a unified interface of dataloader, data augmentor, and data processor for multiple public benchmarks, including Waymo, nuScenes, ONCE, Lyft, and KITTI, etc, which is beneficial to study the transferability and generality of 3D perception models among different datasets. Besides, in order to eliminate the domain gaps between different manufacturers and obtain generalizable representations, 3DTrans has integrated typical unlabeled pre-training techniques for giving a better parameter initialization of the current 3D baseline models. For example, we integrate the PointContrast and SESS to support point cloud-based pre-training task.

• Extensibility for Multiple Models: 3DTrans makes the baseline model have the ability of cross-domain/dataset safe transfer and multi-dataset joint training. Without making major changes of the code and 3D model structure, a single-dataset 3D baseline model can be successfully adapted to an unseen domain or dataset by using our 3DTrans.

• 3DTrans is developed based on OpenPCDet codebase, which can be easily integrated with the models developing using OpenPCDet repository. Thanks for their valuable open-sourcing!

Installation for 3DTrans

You may refer to INSTALL.md for the installation of 3DTrans.

Getting Started

• Please refer to Readme for Datasets to prepare the dataset and convert the data into the 3DTrans format. Besides, 3DTrans supports the reading and writing data from Ceph Petrel-OSS, please refer to Readme for Datasets for more details.

• Please refer to Readme for UDA for understanding the problem definition of UDA and performing the UDA adaptation process.

• Please refer to Readme for ADA for understanding the problem definition of ADA and performing the ADA adaptation process.

• Please refer to Readme for SSDA for understanding the problem definition of SSDA and performing the SSDA adaptation process.

• Please refer to Readme for MDF for understanding the problem definition of MDF and performing the MDF joint-training process.

Model Zoo:

We could not provide the Waymo-related pretrained models due to Waymo Dataset License Agreement, but you could easily achieve similar performance by training with the corresponding configs.

UDA Results:

Here, we report the cross-dataset (Waymo-to-KITTI) adaptation results using the BEV/3D AP performance as the evaluation metric. Please refer to Readme for UDA for experimental results of more cross-domain settings.

• All LiDAR-based models are trained with 4 NVIDIA A100 GPUs and are available for download.
• For Waymo dataset training, we train the model using 20% data.
• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.
• Pre-SN represents that we perform the SN (statistical normalization) operation during the pre-training source-only model stage.
• Post-SN represents that we perform the SN (statistical normalization) operation during the adaptation stage.

ADA Results:

Here, we report the Waymo-to-KITTI adaptation results using the BEV/3D AP performance. Please refer to Readme for ADA for experimental results of more cross-domain settings.

• All LiDAR-based models are trained with 4 NVIDIA A100 GPUs and are available for download.
• For Waymo dataset training, we train the model using 20% data.
• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.

SSDA Results:

We report the target domain results on Waymo-to-nuScenes adaptation using the BEV/3D AP performance as the evaluation metric, and Waymo-to-ONCE adaptation using ONCE evaluation metric. Please refer to Readme for SSDA for experimental results of more cross-domain settings.

• For Waymo-to-nuScenes adaptation, we employ 4 NVIDIA A100 GPUs for model training.
• The domain adaptation time is measured with 4 NVIDIA A100 GPUs and PyTorch 1.8.1.
• For Waymo dataset training, we train the model using 20% data.
• second_5%_FT denotes that we use 5% nuScenes training data to fine-tune the Second model.
• second_5%_SESS denotes that we utilize the SESS: Self-Ensembling Semi-Supervised method to adapt our baseline model.
• second_5%_PS denotes that we fine-tune the source-only model to nuScenes datasets using 5% labeled data, and perform the pseudo-labeling process on the remaining 95% unlabeled nuScenes data.

• For Waymo-to-ONCE adaptation, we employ 8 NVIDIA A100 GPUs for model training.
• PS denotes that we pseudo-label the unlabeled ONCE and re-train the model on pseudo-labeled data.
• SESS denotes that we utilize the SESS method to adapt the baseline.
• For ONCE, the IoU thresholds for evaluation are 0.7, 0.3, 0.5 for Vehicle, Pedestrian, Cyclist.

MDF Results:

Here, we report the Waymo-and-nuScenes consolidation results. The models are jointly trained on Waymo and nuScenes datasets, and evaluated on Waymo using the mAP/mAHPH LEVEL_2 and nuScenes using the BEV/3D AP. Please refer to Readme for MDF for more results.

• All LiDAR-based models are trained with 8 NVIDIA A100 GPUs and are available for download.
• The multi-domain dataset fusion (MDF) training time is measured with 8 NVIDIA A100 GPUs and PyTorch 1.8.1.
• For Waymo dataset training, we train the model using 20% training data for saving training time.
• PV-RCNN-nuScenes represents that we train the PV-RCNN model only using nuScenes dataset, and PV-RCNN-DM indicates that we merge the Waymo and nuScenes datasets and train on the merged dataset. Besides, PV-RCNN-DT denotes the domain attention-aware multi-dataset training.

Point Cloud Pre-training for Autonomous Driving Task

Based on our research progress on the cross-domain adaptation of multiple autonomous driving datasets, we can utilize the multi-source datasets for performing the pre-training task. Here, we present several unsupervised and self-supervised pre-training implementations (including PointContrast).

• Please refer to Readme for point cloud pre-training for starting the journey of 3D perception model pre-training.

• 💪 💪 We are actively exploring the possibility of boosting the 3D pre-training generalization ability. The corresponding code and pre-training checkpoints are coming soon in 3DTrans-v0.2.0.

Visualization Tools for 3DTrans

• Our 3DTrans supports the sequence-level visualization function Quick Sequence Demo to continuously display the prediction results of ground truth of a selected scene.

• Visualization Demo:

  • Waymo Sequence-level Visualization Demo1
  • Waymo Sequence-level Visualization Demo2
  • nuScenes Sequence-level Visualization Demo
  • ONCE Sequence-level Visualization Demo

Acknowledge

• Our code is heavily based on OpenPCDet v0.5.2. Thanks OpenPCDet Development Team for their awesome codebase.

• Our pre-training 3D point cloud task is based on ONCE Dataset. Thanks ONCE Development Team for their inspiring data release.

Our Papers:

• Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection
  
• Bi3D: Bi-domain Active Learning for Cross-domain 3D Object Detection
  
• AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset
  
• SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification
  

Our Team:

• A Team Home for Member Information and Profile, Project Link

Related Projects

• Welcome to see SUG for exploring the possibilities of adapting a model to multiple unseen domains and studying how to leverage the feature’s multi-modal information residing in a single dataset. You can also use SUG to obtain generalizable features by means of a single dataset, which can obtain good results on unseen domains or datasets.

Citation

If you find this project useful in your research, please consider citing:

@misc{3dtrans2023,
    title={3DTrans: An Open-source Codebase for Exploring Transferable Autonomous Driving Perception Task},
    author={3DTrans Development Team},
    howpublished = {\url{https://github.com/PJLab-ADG/3DTrans}},
    year={2023}
}

@inproceedings{zhang2023uni3d,
  title={Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection},
  author={Zhang, Bo and Yuan, Jiakang and Shi, Botian and Chen, Tao and Li, Yikang and Qiao, Yu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9253--9262},
  year={2023}
}

@inproceedings{yuan2023bi3d,
  title={Bi3D: Bi-domain Active Learning for Cross-domain 3D Object Detection},
  author={Yuan, Jiakang and Zhang, Bo and Yan, Xiangchao and Chen, Tao and Shi, Botian and Li, Yikang and Qiao, Yu},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={15599--15608},
  year={2023}
}

@article{yuan2023AD-PT,
  title={AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset},
  author={Yuan, Jiakang and Zhang, Bo and Yan, Xiangchao and Chen, Tao and Shi, Botian and Li, Yikang and Qiao, Yu},
  journal={arXiv preprint arXiv:2306.00612},
  year={2023}
}

@article{huang2023sug,
  title={SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification},
  author={Huang, Siyuan and Zhang, Bo and Shi, Botian and Gao, Peng and Li, Yikang and Li, Hongsheng},
  journal={arXiv preprint arXiv:2305.09160},
  year={2023}
}