OpenOcc: Easily Extendable 3D Occupancy Prediction Codebase
OpenOcc is an open source 3D occupancy prediction codebase implemented with PyTorch.
Highlight Features
-
Multiple Benchmarks Support.
We support training and evaluation on different benchmarks including nuScenes LiDAR Segmentation, SurroundOcc, OpenOccupancy, and 3D Occupancy Prediction Challenge. You can even train with sparse lidar supervision and evaluate with dense annotations. π
-
Extendable Modular Design.
We design our pipeline to be easily composable and extendable. Feel free to explore other combinations like TPVDepth, VoxelDepth, or TPVFusion with simple modifications. π
Demo
Method
Pipeline
Dataset
Status | Name | Description |
---|---|---|
β | ImagePointWrapper | nuScenes LiDAR Segmentation |
β | SurroundOcc | |
β | NuScenes3DOcc | OpenOccupancy |
β | NuScenes3DOPC | 3D Occupancy Prediction Challenge |
2D-3D Lifter
Image2Voxel
Status | Name | Description |
---|---|---|
β | TPVDepthLSSLifter | Use estimated depth distribution to lift image features to the voxel space (LSS). |
β | TPVPlainLSSLifter | Uniformly put image features on the corresponding ray (MonoScene). |
Voxel2Rep
Status | Name | Description |
---|---|---|
β | TPVDepthLSSLifter, TPVPlainLSSLifter | Perform pooling to obtain TPV features. |
β | Perform pooling to obtain BEV features. |
Image2Rep
Status | Name | 3D Scene Representation | Description |
---|---|---|---|
β | TPVQueryLifter | TPV | Use deformable cross-attention to update TPV queries |
β | BEV | Use deformable cross-attention to update BEV queries | |
β | Voxel | Use deformable cross-attention to update Voxel queries |
Encoder
Status | Name | Description |
---|---|---|
β | TPVFormerEncoder | Use self-attention to aggregate features |
β | TPVConvEncoder | Use 2D convolution to aggregate features |
β | Use 3D convolution to aggregate features |
Loss
Status | Name | Description |
---|---|---|
β | CELoss | Cross-entropy loss |
β | LovaszSoftmaxLoss | Lovasz-softmax loss |
Model Zoo
Coming soon.
How to use
Installation
-
Create conda environment with python version 3.8
-
Install pytorch and torchvision with versions specified in requirements.txt
-
Follow instructions in https://mmdetection3d.readthedocs.io/en/latest/getting_started.html#installation to install mmcv-full, mmdet, mmsegmentation, mmdet3d with versions specified in requirements.txt
-
Install timm, numba and pyyaml with versions specified in requirements.txt
-
Install cuda extensions.
python setup.py develop
Preparing
- Download pretrain weights and put them in ckpts/
# ImageNet-1K pretrained ResNet50, same as torchvision://resnet50
https://cloud.tsinghua.edu.cn/f/3d0cea3f6ac24e019cea/?dl=1
- Create soft link from data/nuscenes to your_nuscenes_path. The dataset should be organized as follows:
TPVFormer/data
nuscenes - downloaded from www.nuscenes.org
lidarseg
maps
samples
sweeps
v1.0-trainval
nuscenes_infos_train.pkl
nuscenes_infos_val.pkl
- Download train/val pickle files and put them in data/ nuscenes_infos_train.pkl https://cloud.tsinghua.edu.cn/f/ede3023e01874b26bead/?dl=1 nuscenes_infos_val.pkl https://cloud.tsinghua.edu.cn/f/61d839064a334630ac55/?dl=1
Getting Started
Training
- Train TPVFormer for lidar segmentation task.
bash launcher.sh config/tpvformer/tpvformer_lidarseg_dim128_r50_800.py out/tpvformer_lidarseg_dim128_r50_800
- Train TPVConv with PlainLSSLifter for lidar segmentation task.
bash launcher.sh config/tpvconv/tpvconv_lidarseg_dim384_r50_800_layer10.py out/tpvconv_lidarseg_dim384_r50_800_layer10
- Train TPVConv with DepthLSSLifter for lidar segmentation task.
bash launcher.sh config/tpvconv/tpvconv_lidarseg_dim384_r50_800_layer10_depthlss.py out/tpvconv_lidarseg_dim384_r50_800_layer10_depthlss
HFAI Compatibility
There are only two steps to launch experiments on High-Flyer AI Platform.
Prepare dataset
-
Create soft link from hfai_nuscenes_path to data/nuscenes
-
Download nuScenes-lidarseg-all-v1.0.tar from nuscenes.org, and extract files to data/lidarseg
-
Download maps.tar.gz from https://cloud.tsinghua.edu.cn/f/a74a0dd52bb9459699f2/?dl=1, and extract files to data/maps
-
The final data/ directory should be organized as follows.
OpenOcc/data
nuscenes
lidarseg
lidarseg
v1.0-mini
v1.0-trainval
v1.0-test
maps
*.png
nuscenes_infos_train.pkl
nuscenes_infos_val.pkl
Getting started
Simply add --hfai to your shell command to launch experiments on High-Flyer AI Platform.