gist-ailab/uoais gist-ailab/uoais

  • Stars
    star
    125
  • Rank 286,335 (Top 6 %)
  • Language
    Python
  • License
    Other
  • Created about 3 years ago
  • Updated almost 2 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
gist-ailab/uoais
github

gist-ailab

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Codes of paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling", ICRA 2022

Unseen Object Amodal Instance Segmentation (UOAIS)

Seunghyeok Back, Joosoon Lee, Taewon Kim, Sangjun Noh, Raeyoung Kang, Seongho Bak, Kyoobin Lee

This repository contains source codes for the paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling." (ICRA 2022)

[Paper] [ArXiv] [Project Website] [Video]

Updates & TODO Lists

  • (2021.09.26) UOAIS-Net has been released
  • (2021.11.15) Inference codes for kinect azure and OSD dataset.
  • (2021.11.22) ROS nodes for kinect azure and realsense D435
  • (2021.12.22) Train and evaluation codes on OSD and OCID dataset + OSD-Amodal annotation

Getting Started

Environment Setup

Tested on Titan RTX with python 3.7, pytorch 1.8.0, torchvision 0.9.0, CUDA 10.2 / 11.1 and detectron2 v0.5 / v0.6

  1. Download source codes and checkpoints
git clone https://github.com/gist-ailab/uoais.git
cd uoais
mkdir output

  1. Download checkpoints at GDrive

  2. Move the R50_depth_mlc_occatmask_hom_concat and R50_rgbdconcat_mlc_occatmask_hom_concat to the output folder.

  3. Move the rgbd_fg.pth to the foreground_segmentation folder

  4. Set up a python environment

conda create -n uoais python=3.8
conda activate uoais
pip install torch torchvision 
pip install shapely torchfile opencv-python pyfastnoisesimd rapidfuzz termcolor

  1. Install detectron2

  2. Build custom AdelaiDet inside this repo (at the uoais folder)

python setup.py build develop

Run on sample OSD dataset

# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/run_on_OSD.py --use-cgnet --dataset-path ./sample_data --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/run_on_OSD.py --use-cgnet --dataset-path ./sample_data  --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/run_on_OSD.py --dataset-path ./sample_data --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/run_on_OSD.py --dataset-path ./sample_data --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Run with ROS

  1. Realsense D-435 (realsense-ros is required.)
# launch realsense2 driver
roslaunch realsense2_camera rs_aligned_depth.launch
# launch uoais node
roslaunch uoais uoais_rs_d435.launch 
# or you can use rosrun
rosrun uoais uoais_node.py _mode:="topic"
  1. Azure Kinect (Azure_kinect_ROS_Driver is required)
# launch azure kinect driver
roslaunch azure_kinect_ros_driver driver.launch
# launch uoais node
roslaunch uoais uoais_k4a.launch

Topics & service

  • /uoais/vis_img (sensor_msgs/Image): visualization results
  • /uoais/results (uoais/UOAISResults): UOAIS-Net predictions (mode:=topic)
  • /get_uoais_results (uoais/UOAISRequest): UOAIS-Net predictions (mode:=service)

Parameters

  • mode (string): running mode of ros node (topic or service)
  • rgb (string): topic name of the input rgb
  • depth (string): topic name of the input depth
  • camera_info (string): topic name of the input camera info
  • use_cgnet (bool): use CG-Net [1] for foreground segmentation or not
  • use_planeseg (bool): use RANSAC for plane segmentation or not
  • ransac_threshold (float): max distance a point can be from the plane model

Run without ROS

  1. Realsense D-435 (librealsense and pyrealsense2 are required.)
# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/rs_demo.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/rs_demo.py --use-cgnet --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/rs_demo.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/rs_demo.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
  1. Azure Kinect (Azure-Kinect-Sensor-SDK and pyk4a are required.)
# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python tools/k4a_demo.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
UOAIS-Net (depth) + CG-Net (foreground segmentation)
python tools/k4a_demo.py --use-cgnet --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (RGB-D)
python tools/k4a_demo.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python tools/k4a_demo.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Train & Evaluation

Dataset Preparation

  1. Download UOAIS-Sim.zip and OSD-Amodal-annotations.zip at GDrive
  2. Download OSD-0.2-depth.zip at OSD. [2]
  3. Download OCID dataset at OCID. [3]
  4. Extract the downloaded datasets and organize the folders as follows
uoais
├── output
└── datasets
       ├── OCID-dataset # for evaluation on indoor scenes
       │     └──ARID10
       │     └──ARID20
       │     └──YCB10
       ├── OSD-0.20-depth # for evaluation on tabletop scenes
       │     └──amodal_annotation # OSD-amodal
       │     └──annotation
       │     └──disparity
       │     └──image_color
       │     └──occlusion_annotation # OSD-amodal
       └── UOAIS-Sim # for training
              └──annotations
              └──train
              └──val

Train on UOAIS-Sim

# UOAIS-Net (RGB-D) 
python train_net.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth) 
python train_net.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Evaluation on OSD dataset

# UOAIS-Net (RGB-D) + CG-Net (foreground segmentation)
python eval/eval_on_OSD.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml --use-cgnet
# UOAIS-Net (depth) + CG-Net (foreground segmentation)
python eval/eval_on_OSD.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml --use-cgnet

This code evaluates the UOAIS-Net that was trained on a single seed (7), thus the metrics from this code and the paper (an average of seeds 7, 77, 777) can be different.

Evaluation on OCID dataset

# UOAIS-Net (RGB-D)
python eval/eval_on_OCID.py --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml
# UOAIS-Net (depth)
python eval/eval_on_OCID.py --config-file configs/R50_depth_mlc_occatmask_hom_concat.yaml

Visualization on OSD dataset

python tools/run_on_OSD.py --use-cgnet --config-file configs/R50_rgbdconcat_mlc_occatmask_hom_concat.yaml

License

The source code of this repository is released only for academic use. See the license file for details.

Notes

The codes of this repository are built upon the following open sources. Thanks to the authors for sharing the code!

Citation

If you use our work in a research project, please cite our work:

@inproceedings{back2022unseen,
  title={Unseen object amodal instance segmentation via hierarchical occlusion modeling},
  author={Back, Seunghyeok and Lee, Joosoon and Kim, Taewon and Noh, Sangjun and Kang, Raeyoung and Bak, Seongho and Lee, Kyoobin},
  booktitle={2022 International Conference on Robotics and Automation (ICRA)},
  pages={5085--5092},
  year={2022},
  organization={IEEE}
}

References

[1] SUN, Yao, et al. Cg-net: Conditional gis-aware network for individual building segmentation in vhr sar images. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-15.
[2] Richtsfeld, Andreas, et al. "Segmentation of unknown objects in indoor environments." 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.
[3] Suchi, Markus, et al. "EasyLabel: a semi-automatic pixel-wise object annotation tool for creating robotic RGB-D datasets." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
[4] Xiang, Yu, et al. "Learning rgb-d feature embeddings for unseen object instance segmentation." Conference on Robot Learning (CoRL). 2020.
[5] Xiao, Yuting, et al. "Amodal Segmentation Based on Visible Region Segmentation and Shape Prior." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.
[6] DENNINGER, Maximilian, et al. Blenderproc. arXiv preprint arXiv:1911.01911, 2019.

More Repositories

1

teaching-where-to-look

Official Implementation of the ECCV 22 paper, "Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition".
Python
54
star
2

SleePyCo

This is an official implementation for "SleePyCo: Automatic Sleep Scoring with Feature Pyramid and Contrastive Learning".
Python
41
star
3

SF-Mask-RCNN

Synthetic RGB-D Fusion (SF) Mask R-CNN for Unseen Object Instance Segmentation
Python
28
star
4

IITNet-official

This is an official implementation for "Intra- and inter-epoch temporal context network (IITNet) using sub-epoch features for automatic sleep scoring on raw single-channel EEG".
Python
28
star
5

block-selection-for-OOD-detection

This is an official implementation for "Block Selection Method for Using Feature Norm in Out-of-distribution Detection".
Python
22
star
6

deep-grasping

ROS wrapper for DNN based robotic grasping algorithms
Python
18
star
7

easy-ros-tutorial

ROS Tutorial and Examples: Play with the camera and robot using Python
Python
18
star
8

Food-Instance-Segmentation

Deep learning based food instance segmentation using synthetic dataset. We provide the trainig, evaluate and inference codes. Also the trained weight and dataset generated by blener are free to use
Python
14
star
9

feature-similarity-KD

Official implementation of the paper "Enhancing Low Resolution Face Recognition with Feature Similarity Knowledge Distillation""
Python
13
star
10

uop-net

[RA-L + IROS2024] Learning to place unseen objects stably using large-scale simulation
Python
13
star
11

AILAB-summer-school

Python
11
star
12

easy_ros_tutorial

ROS Tutorial and Examples: Play with the camera and robot using Python
Python
11
star
13

domain-specific-block-selection-and-paired-view-pseudo-labeling-for-online-TTA

An official implementation of "Domain Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptation", CVPR 2024.
Python
10
star
14

MART

This repo is the official implementation of "MART: MultiscAle Relational Transformer Networks for Trajectory Prediction"
9
star
15

mooc

Lecture Notes & Python Codes for MOOC Class
Python
8
star
16

AILAB-isaac-sim-pick-place

Jupyter Notebook
7
star
17

assembly-part-simulation

Python
6
star
18

insta-beer

Codes of paper "Fast and Accurate Unknown Object Instance Segmentation through Error-Informed Refinement"
3
star
19

BattleSound

Jupyter Notebook
3
star
20

Predict_Fire_Propagation

predicting the trajectory of wildfire using deep learning
Python
2
star
21

image-crawler

Python
2
star
22

pose-anno-tool

Object and Hand pose annotation toolkit for occluded dataset project.
Python
2
star
23

process-smd-plus

Python
2
star
24

assembly-camera-manager

camera manager for furniture assembly project (Kinect Azure and Zivid)
Python
2
star
25

samsung-cadquery-tools

Generate CAD data using cadquery library
Python
1
star
26

qbhand-interface

tutorials and examples to control QB softhand with ROS
CMake
1
star
27

PP-PCS

Python
1
star
28

decremental

codes for decremental learning
Python
1
star
29

assembly-kitting-manager

connector segmentation and axis detection for robotic grasping and kitting
Python
1
star
30

dual-arm-grasp-gen

Python
1
star