D-NeRF: Neural Radiance Fields for Dynamic Scenes
[Project] [Paper]
D-NeRF is a method for synthesizing novel views, at an arbitrary point in time, of dynamic scenes with complex non-rigid geometries. We optimize an underlying deformable volumetric function from a sparse set of input monocular views without the need of ground-truth geometry nor multi-view images.
This project is an extension of NeRF enabling it to model dynmaic scenes. The code heavily relays on NeRF-pytorch.
Installation
git clone https://github.com/albertpumarola/D-NeRF.git
cd D-NeRF
conda create -n dnerf python=3.6
conda activate dnerf
pip install -r requirements.txt
cd torchsearchsorted
pip install .
cd ..
Download Pre-trained Weights
You can download the pre-trained models from drive or dropbox. Unzip the downloaded data to the project root dir in order to test it later. See the following directory structure for an example:
โโโ logs
โ โโโ mutant
โ โโโ standup
โ โโโ ...
Download Dataset
You can download the datasets from drive or dropbox. Unzip the downloaded data to the project root dir in order to train. See the following directory structure for an example:
โโโ data
โ โโโ mutant
โ โโโ standup
โ โโโ ...
Demo
We provide simple jupyter notebooks to explore the model. To use them first download the pre-trained weights and dataset.
| Description | Jupyter Notebook |
|---|---|
| Synthesize novel views at an arbitrary point in time. | render.ipynb |
| Reconstruct mesh at an arbitrary point in time. | reconstruct.ipynb |
| Quantitatively evaluate trained model. | metrics.ipynb |
Test
First download pre-trained weights and dataset. Then,
python run_dnerf.py --config configs/mutant.txt --render_only --render_test
This command will run the mutant experiment. When finished, results are saved to ./logs/mutant/renderonly_test_799999 To quantitatively evaluate model run metrics.ipynb notebook
Train
First download the dataset. Then,
conda activate dnerf
export PYTHONPATH='path/to/D-NeRF'
export CUDA_VISIBLE_DEVICES=0
python run_dnerf.py --config configs/mutant.txt
Citation
If you use this code or ideas from the paper for your research, please cite our paper:
@article{pumarola2020d,
title={D-NeRF: Neural Radiance Fields for Dynamic Scenes},
author={Pumarola, Albert and Corona, Enric and Pons-Moll, Gerard and Moreno-Noguer, Francesc},
journal={arXiv preprint arXiv:2011.13961},
year={2020}
}