autonomousvision/convolutional_occupancy_networks autonomousvision/convolutional_occupancy_networks

  • Stars
    star
    792
  • Rank 57,492 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created over 4 years ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
autonomousvision/convolutional_occupancy_networks
github

autonomousvision

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[ECCV'20] Convolutional Occupancy Networks

Convolutional Occupancy Networks

Paper | Supplementary | Video | Teaser Video | Project Page | Blog Post

This repository contains the implementation of the paper:

Convolutional Occupancy Networks
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys and Andreas Geiger
ECCV 2020 (spotlight)

If you find our code or paper useful, please consider citing

@inproceedings{Peng2020ECCV,
 author =  {Peng, Songyou and Niemeyer, Michael and Mescheder, Lars and Pollefeys, Marc and Geiger, Andreas},
 title = {Convolutional Occupancy Networks},
 booktitle = {European Conference on Computer Vision (ECCV)},
 year = {2020}}

Contact Songyou Peng for questions, comments and reporting bugs.

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called conv_onet using

conda env create -f environment.yaml
conda activate conv_onet

Note: you might need to install torch-scatter mannually following the official instruction:

pip install torch-scatter==2.0.4 -f https://pytorch-geometric.com/whl/torch-1.4.0+cu101.html

Next, compile the extension modules. You can do this via

python setup.py build_ext --inplace

Demo

First, run the script to get the demo data:

bash scripts/download_demo_data.sh

Reconstruct Large-Scale Matterport3D Scene

You can now quickly test our code on the real-world scene shown in the teaser. To this end, simply run:

python generate.py configs/pointcloud_crop/demo_matterport.yaml

This script should create a folder out/demo_matterport/generation where the output meshes and input point cloud are stored.

Note: This experiment corresponds to our fully convolutional model, which we train only on the small crops from our synthetic room dataset. This model can be directly applied to large-scale real-world scenes with real units and generate meshes in a sliding-window manner, as shown in the teaser. More details can be found in section 6 of our supplementary material. For training, you can use the script pointcloud_crop/room_grid64.yaml.

Reconstruct Synthetic Indoor Scene

You can also test on our synthetic room dataset by running:

python generate.py configs/pointcloud/demo_syn_room.yaml

Dataset

To evaluate a pretrained model or train a new model from scratch, you have to obtain the respective dataset. In this paper, we consider 4 different datasets:

ShapeNet

You can download the dataset (73.4 GB) by running the script from Occupancy Networks. After, you should have the dataset in data/ShapeNet folder.

Synthetic Indoor Scene Dataset

For scene-level reconstruction, we create a synthetic dataset of 5000 scenes with multiple objects from ShapeNet (chair, sofa, lamp, cabinet, table). There are also ground planes and randomly sampled walls.

You can download our preprocessed data (144 GB) using

bash scripts/download_data.sh

This script should download and unpack the data automatically into the data/synthetic_room_dataset folder.
Note: We also provide point-wise semantic labels in the dataset, which might be useful.

Alternatively, you can also preprocess the dataset yourself. To this end, you can:

  • download the ShapeNet dataset as described above.
  • check scripts/dataset_synthetic_room/build_dataset.py, modify the path and run the code.

Matterport3D

Download Matterport3D dataset from the official website. And then, use scripts/dataset_matterport/build_dataset.py to preprocess one of your favorite scenes. Put the processed data into data/Matterport3D_processed folder.

ScanNet

Download ScanNet v2 data from the official ScanNet website. Then, you can preprocess data with: scripts/dataset_scannet/build_dataset.py and put into data/ScanNet folder.
Note: Currently, the preprocess script normalizes ScanNet data to a unit cube for the comparison shown in the paper, but you can easily adapt the code to produce data with real-world metric. You can then use our fully convolutional model to run evaluation in a sliding-window manner.

Usage

When you have installed all binary dependencies and obtained the preprocessed data, you are ready to run our pre-trained models and train new models from scratch.

Mesh Generation

To generate meshes using a trained model, use

python generate.py CONFIG.yaml

where you replace CONFIG.yaml with the correct config file.

Use a pre-trained model
The easiest way is to use a pre-trained model. You can do this by using one of the config files under the pretrained folders.

For example, for 3D reconstruction from noisy point cloud with our 3-plane model on the synthetic room dataset, you can simply run:

python generate.py configs/pointcloud/pretrained/room_3plane.yaml

The script will automatically download the pretrained model and run the generation. You can find the outputs in the out/.../generation_pretrained folders

Note that the config files are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pretrained model.

We provide the following pretrained models:

pointcloud/shapenet_1plane.pt
pointcloud/shapenet_3plane.pt
pointcloud/shapenet_grid32.pt
pointcloud/shapenet_3plane_partial.pt
pointcloud/shapenet_pointconv.pt
pointcloud/room_1plane.pt
pointcloud/room_3plane.pt
pointcloud/room_grid32.pt
pointcloud/room_grid64.pt
pointcloud/room_combine.pt
pointcloud/room_pointconv.pt
pointcloud_crop/room_grid64.pt
voxel/voxel_shapenet_1plane.pt
voxel/voxel_shapenet_3plane.pt
voxel/voxel_shapenet_grid32.pt

Evaluation

For evaluation of the models, we provide the script eval_meshes.py. You can run it using:

python eval_meshes.py CONFIG.yaml

The script takes the meshes generated in the previous step and evaluates them using a standardized protocol. The output will be written to .pkl/.csv files in the corresponding generation folder which can be processed using pandas.

Note: We follow previous works to use "use 1/10 times the maximal edge length of the current object’s bounding box as unit 1" (see Section 4 - Metrics). In practice, this means that we multiply the Chamfer-L1 by a factor of 10 for reporting the numbers in the paper.

Training

Finally, to train a new network from scratch, run:

python train.py CONFIG.yaml

For available training options, please take a look at configs/default.yaml.

Further Information

Please also check out the following concurrent works that either tackle similar problems or share similar ideas:

More Repositories

1

sdfstudio

A Unified Framework for Surface Reconstruction
Python
1,965
star
2

occupancy_networks

This repository contains the code for the paper "Occupancy Networks - Learning 3D Reconstruction in Function Space"
Python
1,492
star
3

giraffe

This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"
Python
1,227
star
4

stylegan-t

[ICML'23] StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Python
1,122
star
5

mip-splatting

[CVPR'24 Best Student Paper] Mip-Splatting: Alias-free 3D Gaussian Splatting
Python
1,046
star
6

transfuser

[PAMI'23] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving; [CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Python
1,023
star
7

stylegan-xl

[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets
Python
939
star
8

projected-gan

[NeurIPS'21] Projected GANs Converge Faster
Python
876
star
9

unimatch

[TPAMI'23] Unifying Flow, Stereo and Depth Estimation
Python
855
star
10

differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
Python
782
star
11

gaussian-opacity-fields

[SIGGRAPH Asia'24 & TOG] Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes
Python
705
star
12

monosdf

[NeurIPS'22] MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Python
563
star
13

shape_as_points

[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver
Python
518
star
14

tuplan_garage

[CoRL'23] Parting with Misconceptions about Learning-based Vehicle Motion Planning
Python
499
star
15

unisurf

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Python
418
star
16

graf

Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"
Jupyter Notebook
398
star
17

kitti360Scripts

This repository contains utility scripts for the KITTI-360 dataset.
Python
385
star
18

neat

[ICCV'21] NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Python
301
star
19

navsim

[NeurIPS 2024] NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
Python
244
star
20

occupancy_flow

This repository contains the code for the ICCV 2019 paper "Occupancy Flow - 4D Reconstruction by Learning Particle Dynamics"
Python
207
star
21

plant

[CoRL'22] PlanT: Explainable Planning Transformers via Object-Level Representations
Python
201
star
22

factor-fields

[SIGGRAPH 2023] We provide a unified formula for neural fields (Factor Fields) and a novel dictionary factorization (Dictionary Fields)
Jupyter Notebook
183
star
23

sledge

[ECCV'24] SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic
Python
151
star
24

voxgraf

Official code release for VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Python
128
star
25

carla_garage

[ICCV'23] Hidden Biases of End-to-End Driving Models
Python
121
star
26

gta

[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers
Python
121
star
27

texture_fields

This repository contains code for the paper 'Texture Fields: Learning Texture Representations in Function Space'.
Python
115
star
28

kitti360LabelTool

JavaScript
103
star
29

counterfactual_generative_networks

[ICLR'21] Counterfactual Generative Networks
Python
102
star
30

murf

[CVPR'24] MuRF: Multi-Baseline Radiance Fields
Python
84
star
31

king

[ECCV'22] KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients
Python
73
star
32

controllable_image_synthesis

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis, CVPR 2020
Python
70
star
33

handheld_svbrdf_geometry

On Joint Estimation of Pose, Geometry and svBRDF from a Handheld Scanner, CVPR2020
Python
59
star
34

connecting_the_dots

This repository contains the code for the paper "Connecting the Dots: Learning Representations for Active Monocular Depth Estimation" https://avg.is.tuebingen.mpg.de/publications/riegler2019cvpr
Python
56
star
35

frequency_bias

Official code for "On the Frequency Bias of Generative Models", NeurIPS 2021
Python
45
star
36

good

[ICLR'23] GOOD: Exploring Geometric Cues for Detecting Objects in an Open World
Python
39
star
37

data_aggregation

This repository contains the code for the CVPR 2020 paper "Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving"
Python
38
star
38

campari

[3DV'21] CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
Python
29
star
39

akorn

Reproducing code for the work: Artificial Kuramoto Oscillatory Neurons
22
star
40

autonomousvision.github.io

Blog of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
HTML
19
star
41

hdt

[COLM'24] HDT: Hierarchical Document Transformer
Python
7
star
42

visual_abstractions

6
star
43

slides

Slide repository of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
CSS
2
star
44

similarity_reconstruction

This code is based on the paper Exploiting Object Similarity in 3D Reconstruction.
C++
1
star
45

slow_flow

This code is based on the paper Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data.
C++
1
star