DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes

Project page | arXiv

TL;DR: A novel representation and learning paradigm for dynamic radiance fields reconstruction -- 100x faster, no loss in dynamic novel view synthesis quality.

📢 News

• [2022.10.10] We release the first-version of DeVRF code and dataset!

• [2022.9.15] DeVRF got accepted by NeurIPS 2022!

• [2022.6.1] We release the arXiv paper.

📝 Preparation

Installation

git clone https://github.com/showlab/DeVRF.git
cd DeVRF
pip install -r requirements.txt

Pytorch and torch_scatter installation is machine dependent, please install the correct version for your machine.

DeVRF dataset

We release all the synthetic and real-world DeVRF dataset on link. DeVRF dataset consists of 5 inward-facing synthetic scenes (lego|floating_robot|kuka|daisy|glove), 1 inward-facing real-world scene (flower_360), and 3 forward-facing real-world scenes (plant|rabbit|pig_toy). For each scene, we release the static data, dynamic data, and optical flow estimated using RAFT. Please refer to the following data structure for an overview of DeVRF dataset.

    DeVRF dataset
    ├── inward-facing
    │   └── [lego|floating_robot|kuka|daisy|glove|flower_360]
    │       ├── static    
    │       │	├── [train|val|test]
    │       │	└── transforms_[train|val|test].json
    │       └── dynamic_4views    
    │        	├── [train|val|test]
    │        	├── transforms_[train|val|test].json   
    │        	├── train_flow              
    │        	└── train_flow_png  
    │          
    └── forward-facing
        └── [plant|rabbit|pig_toy]
            ├── static    
            │	├── [images|images_4|images_8]
            │	└── poses_bounds.npy
            └── dynamic_4views    
             	├── bds.npy  
             	├── poses.npy                        
             	└── [view1|view2|view3|view4]
	                ├── [images|images_4|images_8]
             	 	├── images_4_flow
             	 	└── images_4_flow_png

We additionally provide a light version of DeVRF dataset without optical flow on link.

🏋️‍️ Experiment

Training

Stage 1: Train the static model using static scene data. The static model part is almost the same as DirectVoxGO. The main difference is that we add an accumulated transmittance loss to encourage a clean background for forward-facing scenes. Please refer to DirectVoxGO for more details.

Note: Please enlarge the world_bound_scale in config file to establish a larger bounding box for dynamic scene modelling in the second stage. For DeVRF dataset, the world_bound_scale parameter is set within [1.05, 2.0].

$ cd static_DirectVoxGO
$ python run.py --config configs/inward-facing/lego.py --render_test

Stage 2: Train the dynamic model using dynamic scene data and the trained static model.

$ cd ..
$ python run.py --config configs/inward-facing/lego.py --render_test

Evaluation

To only evaluate the testset PSNR, SSIM, and LPIPS of the trained lego without re-training, run:

$ python run.py --config configs/inward-facing/lego.py --render_only --render_test \

--eval_ssim --eval_lpips_vgg --eval_lpips_alex

Use --eval_lpips_alex or --eval_lpips_vgg to evaluate LPIPS with pre-trained Alex net or VGG net.

Render video

$ python run.py --config configs/inward-facing/lego.py --render_only --render_video

Reproduction: all config files to reproduce our results.

$ ls configs/*

configs/inward-facing:
lego.py floating_robot.py kuka.py daisy.py glove.py flower_360.py

configs/forward-facing:
plant.py rabbit.py pig_toy.py

🎓 Citation

If you find our work helps, please cite our paper.

@article{liu2022devrf,

title={DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes},

author={Liu, Jia-Wei and Cao, Yan-Pei and Mao, Weijia and Zhang, Wenqiao and Zhang, David Junhao and Keppo, Jussi and Shan, Ying and Qie, Xiaohu and Shou, Mike Zheng},

journal={arXiv preprint arXiv:2205.15723},

year={2022}

}

✉️ Contact

This repo is maintained by Jiawei Liu. Questions and discussions are welcome via [email protected].

🙏 Acknowledgements

This codebase is based on DirectVoxGO.

LICENSE

GPL