valeoai/POCO valeoai/POCO

  • Stars
    star
    178
  • Rank 213,835 (Top 5 %)
  • Language
    Python
  • License
    Other
  • Created over 2 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
valeoai/POCO
github

valeoai

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

POCO: Point Convolution for Surface Reconstruction

by: Alexandre Boulch and Renaud Marlet

Computer Vision and Pattern Recognition, CVPR, 2022

Paper    PDF    Supp    Arxiv

Picture

Abstract

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes.

Citation

@InProceedings{Boulch_2022_CVPR,
    author    = {Boulch, Alexandre and Marlet, Renaud},
    title     = {POCO: Point Convolution for Surface Reconstruction},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {6302-6314}
}

Installation

The code was used with Ubuntu, Python 3.7.10, Cuda 11.1 and Pytorch 1.8.1

Installing APT packages

apt-get install libgl1-mesa-glx libopenblas-dev

Creating a conda environement

If you want to run the code in a conda environment, you can create one with Python 3.7.10 and CudaToolKit 11.1 using:

conda env create -f environment.yml
conda activate poco

Installing dependencies

pip install -r requirements.txt

Building the triangle hash module (from ConvONet)

This module is used for score computation

CC=gcc CXX=gcc python setup_c.py build_ext --inplace
python setup.py build_ext --inplace

Data

ShapeNet (Occupancy Network pre-processing)

We use the ShapeNet dataset as pre-processed by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/ShapeNet.

SyntheticRooms (Occupancy Network pre-processing)

We use the SyntheticRooms dataset as created by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/synthetic_room_dataset.

ABC / RealWorld / Famous and Thingi10k

They should be placed in the folders: data/3d_shapes_abc, data/3d_shapes_abc_training, data/3d_shapes_famous, data/3d_shapes_real_world and data/3d_shapes_thingi10k.

SceneNet

Download the SceneNet dataset.

Watertight models

We use the code from https://github.com/hjwdzh/Manifold for watertight model creation. Please download and compile it.

To create the watertight models, create a destination directory data/SceneNet, copy-paste the python script datasets/scenenet_watertight.py into the directory and run it with python scenenet_watertight.py. You will need to modify the raw_data_dir and the manifold_code_dir to match your data and code locations.

Creation of the point clouds

You can use the script datasets/scenenet_sample.py. You will also need to modify the paths.

Precomputed meshes and point clouds

For easy use of the code, we provide precomputed meshes, and point clouds SceneNet20, SceneNet100, SceneNet500 and SceneNet1000.

Location

The watertight models should be in data/SceneNet.

The points are located in data/SceneNet20, data/SceneNet100, data/SceneNet1000 and data/SceneNet500.

Training

# train on ShapeNet with 3k points, noise and without normals 
python train.py --config configs/config_shapenet.yaml 

# train on ShapeNet with 10k points, no noise and normals
python train.py --config configs/config_shapenet.yaml --normals True --random_noise 0 --experiment_name Normals

# train on SyntheticRooms
python train.py --config configs/config_synthetic.yaml

# train on ABC with 3k points
python train.py --config configs/config_abc.yaml --experiment_name 3k

# train on ABC with 10k points
python train.py --config configs/config_abc.yaml --experiment_name 10k --manifold_points 10000 --training_batch_size 8

Generation

ShapeNet

python generate.py --config results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

SyntheticRooms

python generate.py --config results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 256 --num_mesh 1

ABC, RealWorld, Famous and Thingi10k

# Model trained with 3k points
python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --gen_subsample_manifold 3000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

# Model trained with 10k points
python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --noise 0.0 --gen_subsample_manifold 10000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

For faster generation, one would want to use a lower marching cubes parameter --gen_resolution_global, e.g. 128.

DATASET_NAME should be replaced by ABCTest, ABCTestNoiseFree, ABCTestExtraNoise, RealWorld, FamousTest, FamousTestNoiseFree, FamousTestExtraNoisy, FamousTestSparse, FamousTestDense, Thingi10kTest, Thingi10kTestNoiseFree, Thingi10kTestExtraNoisy, Thingi10kTestSparse, Thingi10kTestDense

DATASET_DIR should be replaced by 3d_shapes_abc, 3d_shapes_famous, 3d_shapes_real_world, 3d_shapes_thingi10k

SceneNet

python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name DATASET_NAME --dataset_root data/DATASET_NAME --manifold_points -1

DATASET_NAME should be replaced by SceneNet20, SceneNet100, SceneNet500, SceneNet1000.

The --gen_autoscale_target parameter is the average distance between a point and its nearest neighbor in the training set, 0.01 for ShapeNet train set with 3000 points.

Evaluation

ShapeNet

python eval_meshes.py --gendir results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/gen_ShapeNet_test_3000/ --meshdir meshes --dataset ShapeNet --split test --gtdir data/ShapeNet

SyntheticRooms

python eval_meshes.py --gendir results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/gen_SyntheticRooms_test_10000/ --meshdir meshes --dataset SyntheticRooms --split test --gtdir data/synthetic_room_dataset

SceneNet

python eval_scenenet.py --gendir results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/gen_SceneNet20_test_allPts/ --meshdir meshes --gtdir data/SceneNet

ABC, Famous and Thingi10k

For evaluation on the ABCTest please run:

python eval_point2surf_datasets.py --gendir results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/gen_ABCTest_test_3000 --meshdir meshes/04_pts/ --gtdir data/3d_shapes_abc/abc/

You can adapt the paths to evaluate on the different datasets and noise variants.

Pretrained models

We provide pre-trained models for FKAConv backbone.

ShapeNet 3k, noise, no normals

ShapeNet 3k, no noise, normals

SyntheticRooms 10k

ABC 3k

ABC 10k

More Repositories

1

WoodScape

The repository containing tools and information about the WoodScape dataset.
Python
602
star
2

ADVENT

Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
Python
379
star
3

LOST

Pytorch implementation of LOST unsupervised object discovery method
Python
234
star
4

xmuda

Cross-Modal Unsupervised Domain Adaptationfor 3D Semantic Segmentation
Python
192
star
5

ZS3

Zero-Shot Semantic Segmentation
Python
187
star
6

SLidR

Official PyTorch implementation of "Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data"
Python
172
star
7

ConfidNet

Addressing Failure Prediction by Learning Model Confidence
Python
162
star
8

ALSO

ALSO: Automotive Lidar Self-supervision by Occupancy estimation
Python
158
star
9

RADIal

Jupyter Notebook
155
star
10

Maskgit-pytorch

Jupyter Notebook
142
star
11

BF3S

Boosting Few-Shot Visual Learning with Self-Supervision
Python
136
star
12

DADA

Depth-aware Domain Adaptation in Semantic Segmentation
Python
114
star
13

FLOT

FLOT: Scene Flow Estimation by Learned Optimal Transport on Point Clouds
Python
95
star
14

obow

Python
95
star
15

carrada_dataset

Jupyter Notebook
84
star
16

rainbow-iqn-apex

Distributed Rainbow-IQN for Atari
Python
75
star
17

rangevit

Python
72
star
18

FOUND

PyTorch code for Unsupervised Object Localization: Observing the Background to Discover Objects
Python
66
star
19

PointBeV

Official implementation of PointBeV: A Sparse Approach to BeV Predictions
Python
63
star
20

LightConvPoint

Python
62
star
21

Awesome-Unsupervised-Object-Localization

Curated list of awesome works on unsupervised object localization in 2D images.
61
star
22

BEVContrast

BEVContrast: Self-Supervision in BEV Space for Automotive Lidar Point Clouds - Official PyTorch implementation
Python
60
star
23

MVRSS

Python
58
star
24

FKAConv

Python
40
star
25

SALUDA

SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation
Python
38
star
26

LaRa

LaRa: Latents and Rays for Multi-Camera Bird’s-Eye-View Semantic Segmentation
Python
38
star
27

WaffleIron

Python
37
star
28

obsnet

Python
32
star
29

BUDA

Boundless Unsupervised Domain Adaptation in Semantic Segmentation
32
star
30

ScaLR

PyTorch code and models for ScaLR image-to-lidar distillation method
Python
32
star
31

3DGenZ

Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"
Python
31
star
32

SemanticPalette

Semantic Palette: Guiding Scene Generation with Class Proportions
Python
29
star
33

xmuda_journal

[TPAMI] Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentation
Python
28
star
34

NeeDrop

NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping
Python
27
star
35

PCAM

Python
27
star
36

MTAF

Multi-Target Adversarial Frameworks for Domain Adaptation in Semantic Segmentation
Python
23
star
37

ESL

ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation
Python
19
star
38

STEEX

STEEX: Steering Counterfactual Explanations with Semantics
Python
18
star
39

OCTET

Python
17
star
40

CAB

Python
16
star
41

MuHDi

Official PyTorch implementation of "Multi-Head Distillation for Continual Unsupervised Domain Adaptation in Semantic Segmentation"
Python
15
star
42

diffhpe

Official code of "DiffHPE: Robust, Coherent 3D Human Pose Lifting with Diffusion"
Python
14
star
43

bravo_challenge

BRAVO Challenge Toolkit and Evaluation Code
Python
14
star
44

sfrik

Official code for "Self-supervised learning with rotation-invariant kernels"
Python
12
star
45

BEEF

Python
11
star
46

SP4ASC

Python
7
star
47

bownet

Learning Representations by Predicting Bags of Visual Words
7
star
48

QuEST

Python
5
star
49

MFEval

[ICRA2024] Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? This is the official implementation of the evaluation protocol proposed in this work for motion forecasting models with real-world perception inputs.
Python
5
star
50

Occfeat

5
star
51

dl_utils

The library used in the Valeo Deep learning training.
Python
3
star
52

tutorial-images

2
star
53

PAFUSE

Official repository of PAFUSE
Python
2
star
54

MOCA

MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments
2
star
55

valeoai.github.io

JavaScript
1
star