• Stars
    star
    250
  • Rank 162,397 (Top 4 %)
  • Language
    Python
  • License
    Other
  • Created over 2 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Multi-View Mesh Reconstruction with Neural Deferred Shading (CVPR 2022)

Neural Deferred Shading

Project Page  |  Paper

alt text

Official code for the CVPR 2022 paper "Multi-View Mesh Reconstruction with Neural Deferred Shading", a method for fast multi-view reconstruction with analysis-by-synthesis.

Installation

Setup the environment and install basic requirements using conda

conda env create -f environment.yml
conda activate nds

Nvdiffrast

To install Nvdiffrast from source, run the following in the main directory:

git clone https://github.com/NVlabs/nvdiffrast.git
cd nvdiffrast
python -m pip install .

pyremesh

Option 1 (preferred): Install pyremesh from pre-built packages in the pyremesh subdirectory.

From the main directory, run:

python -m pip install --no-index --find-links ./ext/pyremesh pyremesh

Option 2: Install pyremesh from source.

Follow the instructions at https://github.com/sgsellan/botsch-kobbelt-remesher-libigl.

Reconstructing DTU Scans

Download the full dataset (2.3 GB) or two samples (300 MB) and unzip the content into the main directory. For example, after unzipping you should have the directory ./data/65_skull.

To start the reconstruction for the skull, run:

python reconstruct.py --input_dir ./data/65_skull/views --input_bbox ./data/65_skull/bbox.txt

or for a general scan:

python reconstruct.py --input_dir ./data/{SCAN-ID}_{SCAN-NAME}/views --input_bbox ./data/{SCAN-ID}_{SCAN-NAME}/bbox.txt

You will find the output meshes in the directory ./out/{SCAN-ID}_{SCAN-NAME}/meshes.

Reconstructing Custom Scenes

Our pipeline expects the input data in a specific structure, which you have to follow for your own scenes.

Views (--input_dir)

The main input is a folder with views, where each view consists of an RGB(A) image and the corresponding camera pose and camera intrinsics. An example folder with N views could look like this (the views do not have to be numbered and can have any file names):

📂views
├─🖼️1.png
├─📜1_k.txt
├─📜1_r.txt
├─📜1_t.txt
⋮
├─🖼️N.png
├─📜N_k.txt
├─📜N_r.txt
└─📜N_t.txt

If present, the alpha channel of the image is used as object mask.

The files ..._k.txt, ..._r.txt, and ..._t.txt contain numpy-readable arrays with the camera pose (R, t) and intrinsics (K) in the standard OpenCV format, so K and R are 3x3 matrices and t is a 3-dimensional column vector, such that

$$ \begin{pmatrix} x & y & 1 \end{pmatrix}^\top \sim \mathbf{K}(\mathbf{R}\begin{pmatrix} X & Y & Z \end{pmatrix}^\top + \mathbf{t}).$$

The image-space coordinates (x, y) are in pixels, so the top left of the image is (x, y) = (0, 0) and the bottom right is (x, y) = (width, height).

Bounding Box (--input_bbox)

Another input to our pipeline is a bounding box of the scene. The bounding box is described by a single text file, which contains a numpy-readable array of size 2x3. The first row has the world space coordinates of the minimum point and the second row those of the maximum point.

For example, if the bounding box is a cube with side length 2 centered at (0, 0, 0), then bbox.txt would simply contain

-1 -1 -1
 1  1  1

Initial Mesh (--initial_mesh)

If you would like to start your reconstruction from a custom initial mesh instead of using one of the pre-defined options, you need to provide its path. The mesh file can have any standard format (obj, ply, ...). We use trimesh for loading, so check their list of supported formats.

Customizing Loading Routines

If you want to tinker with our data loading routines to adapt them to your format, have a look at nds.utils.io.read_views() and nds.core.view.View.load().

Using the Interactive Viewer

We provide an interactive viewer based on OpenGL to inspect the reconstructed meshes and their learned appearance. Before you can launch the viewer, install the additional dependencies with

conda activate nds
pip install glfw==2.5.3 moderngl==5.6.4 pyrr==0.10.3 pyopengl==3.1.6

The pycuda dependency needs to be build from source with OpenGL support. In your preferred directory, run

git clone --recursive https://github.com/inducer/pycuda.git
cd pycuda
git checkout v2022.1

conda activate nds
python ./configure.py --cuda-enable-gl
python setup.py install

The viewer is launched by running the python script view.py, providing the mesh, the neural shader and a bounding box as input. For example, the reconstruction results for the DTU skull can be viewed by running

python .\view.py --mesh .\out\65_skull\meshes\mesh_002000.obj --shader .\out\65_skull\shaders\shader_002000.pt --bbox .\out\65_skull\bbox.txt

Profiling Mode

For the runtime experiments, we added a profiling mode to our reconstruction script that benchmarks individual parts of the code. Since the profiling mode is rather invasive, we have provided it in a separate profiling branch.

The reconstruction can be started in profiling mode by passing the --profile flag to reconstruct.py.

After reconstruction, the output directory will contain the additional file profile.json with the (hierarchical) runtimes.

Citation

If you find this code or our method useful for your academic research, please cite our paper

@InProceedings{worchel:2022:nds,
      author    = {Worchel, Markus and Diaz, Rodrigo and Hu, Weiwen and Schreer, Oliver and Feldmann, Ingo and Eisert, Peter},
      title     = {Multi-View Mesh Reconstruction with Neural Deferred Shading},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2022},
      pages     = {6187-6197}
}

Troubleshooting

CUDA Out of Memory

The reconstruction can be quite heavy on GPU memory and in our experiments we used a GPU with 24 GB.

The memory usage can be reduced by reconstructing with a smaller image resolution. Try passing --image_scale 2 or --image_scale 4 to reconstruct.py, which uses 1/2th or 1/4th of the original resolution. Expect lower memory consumption and better runtime but degraded reconstruction accuracy.

Reconstruction Hangs at Remeshing

While the remeshing step can take some time especially at higher mesh resolutions, it sometimes hangs indefinitely. This issue comes from calling the function remesh_botsch in the pyremesh package, which does not return.

For now, the reconstruction has to be aborted and restarted.

More Repositories

1

vvenc

VVenC, the Fraunhofer Versatile Video Encoder
C++
943
star
2

vvdec

VVdeC, the Fraunhofer Versatile Video Decoder
C++
454
star
3

omaf.js

HTML5 MSE Playback of MPEG 360 VR Tiled Streaming
JavaScript
117
star
4

Self-Organizing-Gaussians

[ECCV '24] Compressing 3D Gaussian Splats by placing their parameters into a 2D grid with local smoothness
Python
110
star
5

QuaDRiGa

QuaDRiGa, short for QUAsi Deterministic RadIo channel GenerAtor, is used for generating realistic radio channel impulse responses for system-level simulations of mobile radio networks.
MATLAB
92
star
6

X-maps

[CVPRW '23] Real-time depth estimation with an event camera and a laser projector
Python
74
star
7

nncodec

Fraunhofer Neural Network Encoder/Decoder (NNCodec)
Python
68
star
8

DeepCABAC

C++
58
star
9

gaussian_gan_decoder

[CVPRW '24] Turning the output of NeRF based 3D GANs into 3DGS scenes in a single forward pass.
Python
54
star
10

D-NeuS

Recovering Fine Details for Neural Implicit Surface Reconstruction (WACV2023)
Python
48
star
11

vvdecWebPlayer

A minimal proof-of-concept VVC player for the web using WebAssembly
JavaScript
37
star
12

casapose

CASAPose: Class-Adaptive and Semantic-Aware Multi-Object Pose Estimation (BMVC 2022)
Python
32
star
13

xpsnr

A filter plug-in adding support for Extended Perceptually Weighted Peak Signal-to-Noise Ratio (XPSNR) measurements in FFmpeg
C
28
star
14

spvloc

[ECCV 2024] SPVLoc: Semantic Panoramic Viewport Matching for 6D Camera Localization in Unseen Environments
Python
22
star
15

pred6dof

Code for our ACMM'20 paper "Kalman Filter-based Head Motion Prediction for Cloud-based Mixed Reality"
TeX
20
star
16

BerlinV2X

Support code and documentation for the Berlin V2X dataset
Jupyter Notebook
18
star
17

pytorch2tikz

Generate Tikz figures from pytorch models
Python
13
star
18

PLAS

Parallel Linear Assignment Sorting - Sorting multi-dimensional data into locally smooth 2D grids.
Python
11
star
19

ai4mobile-industrial

Support code and documentation for the AI4Mobile industrial datasets: iV2V and iV2I+
Jupyter Notebook
5
star
20

BTSeg

BarlowTwin Segementation
Python
4
star
21

DistSensIQE

Models for our paper on "Estimation of distortion sensitivity for visual quality prediction using a convolutional neural network".
Python
3
star
22

CuriouslyEffectiveIQE

Reference implementation for our paper "Curiously Effective Features for Image Quality Prediction"
Python
1
star
23

SOLOPlanes

Python
1
star
24

sidelink-mcs-measurements

Measurement data for the paper "From Empirical Measurements to Augmented Data Rates: A Machine Learning Approach for MCS Adaptation in Sidelink Communication"
1
star
25

libapsm

Cuda
1
star