Templates for 3D Object Pose Estimation Revisited:
Generalization to New objects and Robustness to Occlusions
Van Nguyen Nguyen Yinlin Hu Yang Xiao Mathieu Salzmann Vincent Lepetit
If our project is helpful for your research, please consider citing :
@inproceedings{nguyen2022template,
title={Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions},
author={Nguyen, Van Nguyen and Hu, Yinlin and Xiao, Yang and Salzmann, Mathieu and Lepetit, Vincent},
booktitle={Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
year={2022}}You can also put a star
If you like this project, check out related works from our group:
- NOPE: Novel Object Pose Estimation from a Single Image (arXiv 2023)
- PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6DoF Tracking (3DV 2022)
- BOP visualization toolkit
Updates (WIP)
We have introduced additional features and updates to the codebase:
- Adding wandb logger: training loggers, testing loggers
- Cropping in LINEMOD settings is done with input bounding boxes (there is also predicted in-plane rotation)
- Releasing synthetic templates with Pyrender for faster rendering
- Releasing ready-to-use universal model pretrained on different datasets of BOP challenge HomebrewedDB, HOPE, RU-APC, IC-BIN, IC-MI, TUD-L, T-LESS
- Adding code to generate poses (OpenCV coordinate) from icosahedron with Blender
- Parsing with hydra library, simplifying training_step, testing_step with pytorch lightning
- Path structure (of pretrained models, dataset) is defined as in our recent project NOPE
Click to expand
$ROOT_DIR
├── datasets
├── linemod
├── models
├── test
├── tless
├── ruapc
├── ...
├── templates
├── pretrained
├── moco_v2_800ep_pretrain.pth
├── results
├── experiment1
├── wandb
├── checkpoint
├── experiment2git checkout 50a1087
This repository is running with the Weight and Bias logger. Ensure that you update this user's configuration before conducting any experiments.
Installation 👷
Click to expand
1. Create conda environment
conda env create -f environment.yml
conda activate template
# require only for evaluation: pytorch3d 0.7.0
git clone https://github.com/facebookresearch/pytorch3d.git
python -m pip install -e .
2. Datasets
First, create template poses from icosahedron:
blenderproc run src/poses/create_poses.py
Next, download and process BOP datasets
./src/scripts/download_and_process_datasets.sh
There are two options for the final step (rendering synthetic templates from CAD models):
Option 1: Download pre-rendered synthetic templates:
python -m src.scripts.download_prerendered_templates
Optional: This pre-rendered template set can be manually downloaded from here (12GB).
Option 2: Rendering synthetic templates from scratch (this will take around 1 hour with Nvidia V100)
./src/scripts/render_pyrender_all.sh
Click to expand
It is important to verify that all the datasets are correctly downloaded and processed. For example, by counting the number of images of each folder:
for dir in $ROOT_DIR/datasets/*
do
echo ${dir}
find ${dir} -name "*.png" | wc -l
done
If everything is fine, here are the number of images that you should get:
├── $ROOT_DIR/datasets
├── hb # 55080
├── hope # 1968
├── icbin # 19016
├── icmi # 31512
├── lm # 49822
├── olm # 4856
├── ruapc # 143486
├── tless # 309600
├── tudl # 153152
├── templates (12GB) # 84102Launch a training 🚀
Click to expand
0. (Optional) We use pretrained weight from MoCo v2. You can download it from here or run:
python -m src.scripts.download_moco_weights
If you don't want to use pretrained weights, you can remove the path in this line.
1. Training on all BOP datasets except LINEMOD and T-LESS (only objects 19-30)
python train.py name_exp=train_all
The parsing is done with Hydra library. You can override anything in the configuration by passing arguments. For example:
# experiment 1: change batch_size, using data augmentation, update name_exp
python train.py machine.batch_size=2 use_augmentation=True name_exp=train_augmentation
# experiment 2: change batch_size, using data augmentation, update name_exp, update_lr
python train.py machine.batch_size=2 use_augmentation=True model.lr=0.001 name_exp=train_augmentation_lr0.001
Please check out this training loggers to see how the training loss looks like.
Reproduce quantitative results
Please note that all testing objects are unseen during training!
Click to expand
0. You can download it from here or run:
python -m src.scripts.download_checkpoint
TODO: This is not the final checkpoint. We will update it soon.
1. LINEMOD's objects
python test_lm.py name_exp=test_lm model.checkpoint_path=$CHECKPOINT_PATH
2. TLESS's objects
python test_tless.py name_exp=test_tless model.checkpoint_path=$CHECKPOINT_PATH
Please check out this testing loggers to see how the retrieved results looks like.
Inference on custom objects (WIP)
python gradio_demo.py
Acknowledgement
The code is adapted from Nope, Temos, Unicorn, PoseContrast, CosyPose and BOP Toolkit.
The authors thank Martin Sundermeyer, Paul Wohlhart and Shreyas Hampali for their fast reply, feedback!
Contact
If you have any question, feel free to create an issue or contact the first author at [email protected]
TODO
- Update checkpoints
- Tutorial of training/testing on custom datasets
- Gradio demo with similarity visualization
- Release universal pretrained models