dvlab-research/PointGroup dvlab-research/PointGroup

  • Stars
    star
    376
  • Rank 113,810 (Top 3 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated 4 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
dvlab-research/PointGroup
github

dvlab-research

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation

PointGroup

PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (CVPR2020)

overview

Code for the paper PointGroup:Dual-Set Point Grouping for 3D Instance Segmentation, CVPR 2020 (Oral).

Authors: Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia

[arxiv] [video]

Introduction

Instance segmentation is an important task for scene understanding. Compared to the fully-developed 2D, 3D instance segmentation for point clouds have much room to improve. In this paper, we present PointGroup, a new end-to-end bottom-up architecture, specifically focused on better grouping the points by exploring the void space between objects. We design a two-branch network to extract point features and predict semantic labels and offsets, for shifting each point towards its respective instance centroid. A clustering component is followed to utilize both the original and offset-shifted point coordinate sets, taking advantage of their complementary strength. Further, we formulate the ScoreNet to evaluate the candidate instances, followed by the Non-Maximum Suppression (NMS) to remove duplicates.

Installation

Requirements

  • Python 3.7.0
  • Pytorch 1.1.0
  • CUDA 9.0

Virtual Environment

conda create -n pointgroup python==3.7
source activate pointgroup

Install PointGroup

(1) Clone the PointGroup repository.

git clone https://github.com/llijiang/PointGroup.git --recursive 
cd PointGroup

(2) Install the dependent libraries.

pip install -r requirements.txt
conda install -c bioconda google-sparsehash

(3) For the SparseConv, we apply the implementation of spconv. The repository is recursively downloaded at step (1). We use the version 1.0 of spconv.

Note: We further modify spconv\spconv\functional.py to make grad_output contiguous. Make sure you use our modified spconv.

  • To compile spconv, firstly install the dependent libraries.
conda install libboost
conda install -c daleydeng gcc-5 # need gcc-5.4 for sparseconv

Add the $INCLUDE_PATH$ that contains boost in lib/spconv/CMakeLists.txt. (Not necessary if it could be found.)

include_directories($INCLUDE_PATH$)
  • Compile the spconv library.
cd lib/spconv
python setup.py bdist_wheel
  • Run cd dist and use pip to install the generated .whl file.

(4) Compile the pointgroup_ops library.

cd lib/pointgroup_ops
python setup.py develop

If any header files could not be found, run the following commands.

python setup.py build_ext --include-dirs=$INCLUDE_PATH$
python setup.py develop

$INCLUDE_PATH$ is the path to the folder containing the header files that could not be found.

Data Preparation

(1) Download the ScanNet v2 dataset.

(2) Put the data in the corresponding folders.

  • Copy the files [scene_id]_vh_clean_2.ply, [scene_id]_vh_clean_2.labels.ply, [scene_id]_vh_clean_2.0.010000.segs.json and [scene_id].aggregation.json into the dataset/scannetv2/train and dataset/scannetv2/val folders according to the ScanNet v2 train/val split.

  • Copy the files [scene_id]_vh_clean_2.ply into the dataset/scannetv2/test folder according to the ScanNet v2 test split.

  • Put the file scannetv2-labels.combined.tsv in the dataset/scannetv2 folder.

The dataset files are organized as follows.

PointGroup
├── dataset
│   ├── scannetv2
│   │   ├── train
│   │   │   ├── [scene_id]_vh_clean_2.ply & [scene_id]_vh_clean_2.labels.ply & [scene_id]_vh_clean_2.0.010000.segs.json & [scene_id].aggregation.json
│   │   ├── val
│   │   │   ├── [scene_id]_vh_clean_2.ply & [scene_id]_vh_clean_2.labels.ply & [scene_id]_vh_clean_2.0.010000.segs.json & [scene_id].aggregation.json
│   │   ├── test
│   │   │   ├── [scene_id]_vh_clean_2.ply 
│   │   ├── scannetv2-labels.combined.tsv

(3) Generate input files [scene_id]_inst_nostuff.pth for instance segmentation.

cd dataset/scannetv2
python prepare_data_inst.py --data_split train
python prepare_data_inst.py --data_split val
python prepare_data_inst.py --data_split test

Training

CUDA_VISIBLE_DEVICES=0 python train.py --config config/pointgroup_run1_scannet.yaml

You can start a tensorboard session by

tensorboard --logdir=./exp --port=6666

Inference and Evaluation

(1) If you want to evaluate on validation set, prepare the .txt instance ground-truth files as the following.

cd dataset/scannetv2
python prepare_data_inst_gttxt.py

Make sure that you have prepared the [scene_id]_inst_nostuff.pth files before.

(2) Test and evaluate.

a. To evaluate on validation set, set split and eval in the config file as val and True. Then run

CUDA_VISIBLE_DEVICES=0 python test.py --config config/pointgroup_run1_scannet.yaml

An alternative evaluation method is to set save_instance as True, and evaluate with the ScanNet official evaluation script.

b. To run on test set, set (split, eval, save_instance) as (test, False, True). Then run

CUDA_VISIBLE_DEVICES=0 python test.py --config config/pointgroup_run1_scannet.yaml

c. To test with a pretrained model, run

CUDA_VISIBLE_DEVICES=0 python test.py --config config/pointgroup_default_scannet.yaml --pretrain $PATH_TO_PRETRAIN_MODEL$

Pretrained Model

We provide a pretrained model trained on ScanNet v2 dataset. Download it here. Its performance on ScanNet v2 validation set is 35.2/57.1/71.4 in terms of mAP/mAP50/mAP25.

Visualize

To visualize the point cloud, you should first install mayavi. Then you could visualize by running

cd util 
python visualize.py --data_root $DATA_ROOT$ --result_root $RESULT_ROOT$ --room_name $ROOM_NAME$ --room_split $ROOM_SPLIT$ --task $TASK$

The visualization task could be input, instance_gt, instance_pred, semantic_pred and semantic_gt.

Results on ScanNet Benchmark

Quantitative results on ScanNet test set at the submisison time. scannet_result

TODO List

  • Distributed multi-GPU training

Citation

If you find this work useful in your research, please cite:

@article{jiang2020pointgroup,
  title={PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation},
  author={Jiang, Li and Zhao, Hengshuang and Shi, Shaoshuai and Liu, Shu and Fu, Chi-Wing and Jia, Jiaya},
  journal={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

Acknowledgement

This repo is built upon several repos, e.g., SparseConvNet, spconv and ScanNet.

Contact

If you have any questions or suggestions about this repo, please feel free to contact me ([email protected]).

More Repositories

1

MGM

Official repo for "Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models"
Python
3,111
star
2

LongLoRA

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)
Python
2,563
star
3

LISA

Project Page for "LISA: Reasoning Segmentation via Large Language Model"
Python
1,680
star
4

VoxelNeXt

VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (CVPR 2023)
Python
692
star
5

LLaMA-VID

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (ECCV 2024)
Python
656
star
6

DeepUPE

Underexposed Photo Enhancement Using Deep Illumination Estimation
Python
566
star
7

3D-Box-Segment-Anything

We extend Segment Anything to 3D perception by combining it with VoxelNeXt.
Jupyter Notebook
524
star
8

LLMGA

This project is the official implementation of 'LLMGA: Multimodal Large Language Model based Generation Assistant', ECCV2024
Python
424
star
9

ControlNeXt

Controllable video and image Generation, SVD, Animate Anyone, ControlNet, LoRA
Python
417
star
10

PanopticFCN

Fully Convolutional Networks for Panoptic Segmentation (CVPR2021 Oral)
Python
391
star
11

3DSSD

3DSSD: Point-based 3D Single Stage Object Detector (CVPR 2020)
Python
375
star
12

Video-P2P

Video-P2P: Video Editing with Cross-attention Control
Python
365
star
13

FocalsConv

Focal Sparse Convolutional Networks for 3D Object Detection (CVPR 2022, Oral)
Python
364
star
14

Stratified-Transformer

Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)
Python
362
star
15

DSGN

DSGN: Deep Stereo Geometry Network for 3D Object Detection (CVPR 2020)
Python
324
star
16

PFENet

PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation (TPAMI).
Python
307
star
17

SphereFormer

The official implementation for "Spherical Transformer for LiDAR-based 3D Recognition" (CVPR 2023).
Python
300
star
18

GridMask

Python
281
star
19

ReviewKD

Distilling Knowledge via Knowledge Review, CVPR 2021
Python
249
star
20

Parametric-Contrastive-Learning

Parametric Contrastive Learning (ICCV2021) & GPaCo (TPAMI 2023)
Python
237
star
21

Step-DPO

Implementation for "Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs"
Python
232
star
22

Simple-SR

Include MuCAN, LAPAR, etc.
Python
224
star
23

UVTR

Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)
Python
224
star
24

Facelet_Bank

Facelet-Bank for Fast Portrait Manipulation (pytorch)
Python
208
star
25

SA-AutoAug

Scale-aware Automatic Augmentation for Object Detection (CVPR 2021)
Python
196
star
26

LargeKernel3D

LargeKernel3D: Scaling up Kernels in 3D Sparse CNNs (CVPR 2023)
Python
189
star
27

SNR-Aware-Low-Light-Enhance

This is the official implementation for the paper "SNR-aware low-light image enhancement" in CVPR2022
Python
160
star
28

MASA-SR

MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution (CVPR2021)
Python
158
star
29

ECCV22-P3AFormer-Tracking-Objects-as-Pixel-wise-Distributions

The official code for our ECCV22 oral paper: tracking objects as pixel-wise distributions.
Python
158
star
30

Context-Aware-Consistency

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)
Python
155
star
31

SparseTransformer

A fast and memory-efficient libarary for sparse transformer with varying token numbers (e.g., 3D point cloud).
Python
153
star
32

spconv-plus

Python
152
star
33

EfficientNeRF

The official code for "Efficient Neural Radiance Fields" in CVPR2022.
Python
152
star
34

MiSLAS

Improving Calibration for Long-Tailed Recognition (CVPR2021)
Python
144
star
35

RIVAL

[NeurIPS 2023 Spotlight] Real-World Image Variation by Aligning Diffusion Inversion Chain
Python
143
star
36

MOOD

Official PyTorch implementation of MOOD series: (1) MOODv1: Rethinking Out-of-distributionDetection: Masked Image Modeling Is All You Need. (2) MOODv2: Masked Image Modeling for Out-of-Distribution Detection.
Python
133
star
37

outpainting_srn

Wide-Context Semantic Image Extrapolation, CVPR2019
Python
131
star
38

MSAD

Multi-Scale Aligned Distillation for Low-Resolution Detection (CVPR2021)
Python
127
star
39

DeepVision3D

DeepVision3D is an open source toolbox for point-cloud understanding.
Python
119
star
40

Ref-NPR

[CVPR 2023] Ref-NPR: Reference-Based Non-PhotoRealistic Radiance Fields
Python
119
star
41

VFIformer

Video Frame Interpolation with Transformer (CVPR2022)
Python
112
star
42

Prompt-Highlighter

[CVPR 2024] Prompt Highlighter: Interactive Control for Multi-Modal LLMs
Python
112
star
43

VFF

Voxel Field Fusion for 3D Object Detection (CVPR2022)
Python
95
star
44

SMR

Self-Supervised 3D Mesh Reconstruction from Single Images (CVPR2021)
Python
91
star
45

SCGAN

The implementation of 'Image synthesis via semantic composition', ICCV2021.
Python
81
star
46

Imbalanced-Learning

Imbalanced learning tool for imbalanced recognition and segmentation
Python
79
star
47

JigsawClustering

This is the code for CVPR 2021 oral paper: Jigsaw Clustering for Unsupervised Visual Representation Learning
Python
78
star
48

AttenNorm

Attentive Normalization for Conditional Image Generation
Python
71
star
49

GFS-Seg

The official implementation of Generalized Few-shot Semantic Segmentation (CVPR 2022)
Python
63
star
50

Mask-Attention-Free-Transformer

Official Implementation for "Mask-Attention-Free Transformer for 3D Instance Segmentation"
Python
59
star
51

MoTCoder

This is the official code repository of MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks.
Python
58
star
52

SDSD

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)
Python
48
star
53

GroupContrast

[CVPR 2024] GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding
42
star
54

ProposeReduce

Video Instance Segmentation with a Propose-Reduce Paradigm (ICCV 2021)
Python
41
star
55

Robust-Semantic-Segmentation

Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation （ICCV2021）
Python
40
star
56

Mr-Ben

This is the repo for our paper "Mr-Ben: A Comprehensive Meta-Reasoning Benchmark for Large Language Models"
Python
38
star
57

BAL

BAL: Balancing Diversity and Novelty for Active Learning - Official Pytorch Implementation
Python
38
star
58

TriVol

The official code of TriVol in CVPR-2023
Python
37
star
59

MR-GSM8K

Challenge LLMs to Reason About Reasoning: A Benchmark to Unveil Cognitive Depth in LLMs
Python
37
star
60

DecoupleNet

Official implementation for our ECCV 2022 paper "DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentation"
Python
36
star
61

Dsig

Deep Structured Instance Graph for Distilling Object Detectors (ICCV 2021)
Python
35
star
62

LBGAT

Learnable Boundary Guided Adversarial Training (ICCV2021)
Python
33
star
63

Q-LLM

This is the official repo of "QuickLLaMA: Query-aware Inference Acceleration for Large Language Models"
Python
31
star
64

AGSS-VOS

AGSS-VOS: Attention Guided Single-Shot Video Object Segmentation
Python
20
star
65

MAT

MAT: Mask-Aware Transformer for Large Hole Image Inpainting
Python
16
star
66

MSN

Memory Selection Network for Video Propagation (ECCV 2020)
Python
6
star
67

APD

Python
5
star
68

Point2Pix

The official code of Point2pix in CVPR-2023
2
star
69

TagCLIP

Python
2
star