autonomousvision/factor-fields autonomousvision/factor-fields

  • Stars
    star
    183
  • Rank 205,064 (Top 5 %)
  • Language
    Jupyter Notebook
  • License
    MIT License
  • Created about 1 year ago
  • Updated 10 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
autonomousvision/factor-fields
github

autonomousvision

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[SIGGRAPH 2023] We provide a unified formula for neural fields (Factor Fields) and a novel dictionary factorization (Dictionary Fields)

Factor Fields

Project page | Paper

This repository contains a pytorch implementation for the paper: Factor Fields: A Unified Framework for Neural Fields and Beyond and Dictionary Fields: Learning a Neural Basis Decomposition. Our work present a novel framework for modeling and representing signals, we have also observed that Dictionary Fields offer benefits such as improved approximation quality, compactness, faster training speed, and the ability to generalize to unseen images and 3D scenes.

Installation

Tested on Ubuntu 20.04 + Pytorch 1.13.0

Install environment:

conda create -n FactorFields python=3.9
conda activate FactorFields
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.7 -c pytorch -c nvidia
pip install -r requirements.txt

Optionally install tiny-cuda-nn, only needed if you want to run hash grid based representations.

conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Quick Start

Please ensure that you download the corresponding dataset and extract its contents into the data folder.

Image

The training script can be found at scripts/2D_regression.ipynb, and the configuration file is located at configs/image.yaml.

Girl with a Pearl Earring

SDF

The training script can be found at scripts/sdf_regression.ipynb, and the configuration file is located at configs/sdf.yaml.

GIF

NeRF

The training script can be found at train_per_scene.py:

python train_per_scene.py configs/nerf.yaml defaults.expname=lego dataset.datadir=./data/nerf_synthetic/lego

<img src="https://github.com/apchenstu/GIFs/blob/main/FactorField-mic.gif" alt="GIF" width="500px"

Generalization Image

The training script can be found at 2D_set_regression.ipynb

Inpainting

Generalization NeRF

python train_across_scene.py configs/nerf_set.yaml

GIF

More examples

Command explanation with a nerf example:

  • model.basis_dims=[4, 4, 4, 2, 2, 2] adjusts the number of levels and channels at each level, with a total of 6 levels and 18 channels.
  • model.basis_resos=[32, 51, 70, 89, 108, 128] represents the resolution of the feature embeddings.
  • model.freq_bands=[2.0, 3.2, 4.4, 5.6, 6.8, 8.0] indicates the frequency parameters applied at each level of the coordinate transformation function.
  • model.coeff_type represents the coefficient field representations and can be one of the following: [none, x, grid, mlp, vec, cp, vm].
  • model.basis_type represents the basis field representation and can be one of the following: [none, x, grid, mlp, vec, cp, vm, hash].
  • model.basis_mapping represents the coordinate transformation and can be one of the following: [x, triangle, sawtooth, trigonometric]. Please note that if you want to use orthogonal projection, choose the cp or vm basis type, as they automatically utilize the orthogonal projection functions.
  • model.total_params controls the total model size. It is important to note that the model's size capability is determined by model.basis_resos and model.basis_dims. The total_params parameter mainly affects the capability of the coefficients.
  • exportation.render_only you can rendering item after training by setting this label to 1. Please also specify the defaults.ckpt label.
  • exportation.... you can specify whether to render the items of [render_test, render_train, render_path, export_mesh] after training by enable the corressponding label to 1.

Some pre-defined configurations (such as occNet, DVGO, nerf, iNGP, EG3D) can be found in README_FactorField.py.

COPY RIGHT

Citation

If you find our code or paper helpful, please consider citing both of these papers:

@article{Chen2023factor,
  title={Factor Fields: A Unified Framework for Neural Fields and Beyond},
  author={Chen, Anpei and Xu, Zexiang and Wei, Xinyue and Tang, Siyu and Su, Hao and Geiger, Andreas},
  journal={arXiv preprint arXiv:2302.01226},
  year={2023}
}

@article{Chen2023SIGGRAPH, 
 title={{Dictionary Fields: Learning a Neural Basis Decomposition}}, 
 author={Anpei, Chen and Zexiang, Xu and Xinyue, Wei and Siyu, Tang and Hao, Su and Andreas, Geiger}, 
 booktitle={International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH)}, 
 year={2023}}

More Repositories

1

sdfstudio

A Unified Framework for Surface Reconstruction
Python
1,861
star
2

occupancy_networks

This repository contains the code for the paper "Occupancy Networks - Learning 3D Reconstruction in Function Space"
Python
1,454
star
3

giraffe

This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"
Python
1,227
star
4

stylegan-t

[ICML'23] StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis
Python
1,122
star
5

transfuser

[PAMI'23] TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving; [CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Python
957
star
6

stylegan-xl

[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets
Python
939
star
7

projected-gan

[NeurIPS'21] Projected GANs Converge Faster
Python
876
star
8

unimatch

[TPAMI'23] Unifying Flow, Stereo and Depth Estimation
Python
855
star
9

convolutional_occupancy_networks

[ECCV'20] Convolutional Occupancy Networks
Python
792
star
10

differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
Python
782
star
11

mip-splatting

[CVPR'24 Oral] Mip-Splatting: Alias-free 3D Gaussian Splatting
Python
700
star
12

monosdf

[NeurIPS'22] MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Python
535
star
13

shape_as_points

[NeurIPS'21] Shape As Points: A Differentiable Poisson Solver
Python
518
star
14

unisurf

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction
Python
410
star
15

graf

Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"
Jupyter Notebook
393
star
16

tuplan_garage

[CoRL'23] Parting with Misconceptions about Learning-based Vehicle Motion Planning
Python
370
star
17

kitti360Scripts

This repository contains utility scripts for the KITTI-360 dataset.
Python
353
star
18

neat

[ICCV'21] NEAT: Neural Attention Fields for End-to-End Autonomous Driving
Python
291
star
19

gaussian-opacity-fields

Gaussian Opacity Fields for Efficient and Compact Surface Reconstruction in Unbounded Scenes
Python
285
star
20

occupancy_flow

This repository contains the code for the ICCV 2019 paper "Occupancy Flow - 4D Reconstruction by Learning Particle Dynamics"
Python
207
star
21

plant

[CoRL'22] PlanT: Explainable Planning Transformers via Object-Level Representations
Python
192
star
22

voxgraf

Official code release for VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
Python
123
star
23

carla_garage

[ICCV'23] Hidden Biases of End-to-End Driving Models
Python
121
star
24

texture_fields

This repository contains code for the paper 'Texture Fields: Learning Texture Representations in Function Space'.
Python
113
star
25

sledge

SLEDGE: Synthesizing Simulation Environments for Driving Agents with Generative Models
105
star
26

kitti360LabelTool

JavaScript
103
star
27

counterfactual_generative_networks

[ICLR'21] Counterfactual Generative Networks
Python
102
star
28

gta

[ICLR'24] GTA: A Geometry-Aware Attention Mechanism for Multi-view Transformers
Python
95
star
29

murf

[CVPR'24] MuRF: Multi-Baseline Radiance Fields
Python
84
star
30

controllable_image_synthesis

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis, CVPR 2020
Python
69
star
31

king

[ECCV'22] KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients
Python
61
star
32

handheld_svbrdf_geometry

On Joint Estimation of Pose, Geometry and svBRDF from a Handheld Scanner, CVPR2020
Python
57
star
33

navsim

NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation
Python
52
star
34

connecting_the_dots

This repository contains the code for the paper "Connecting the Dots: Learning Representations for Active Monocular Depth Estimation" https://avg.is.tuebingen.mpg.de/publications/riegler2019cvpr
Python
51
star
35

frequency_bias

Official code for "On the Frequency Bias of Generative Models", NeurIPS 2021
Python
39
star
36

data_aggregation

This repository contains the code for the CVPR 2020 paper "Exploring Data Aggregation in Policy Learning for Vision-based Urban Autonomous Driving"
Python
38
star
37

good

[ICLR'23] GOOD: Exploring Geometric Cues for Detecting Objects in an Open World
Python
36
star
38

campari

[3DV'21] CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
Python
29
star
39

autonomousvision.github.io

Blog of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
HTML
19
star
40

visual_abstractions

6
star
41

slides

Slide repository of the Autonomous Vision Group at MPI-IS Tübingen and University of Tübingen.
CSS
2
star
42

similarity_reconstruction

This code is based on the paper Exploiting Object Similarity in 3D Reconstruction.
C++
1
star
43

slow_flow

This code is based on the paper Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data.
C++
1
star