Representing Volumetric Videos as Dynamic MLP Maps
Project Page | Video | Paper | Data
Representing Volumetric Videos as Dynamic MLP Maps
Sida Peng*, Yunzhi Yan*, Qing Shuai, Hujun Bao, Xiaowei Zhou (* equal contribution)
CVPR 2023
Any questions or discussions are welcomed!
Installation
Please see INSTALL.md for manual installation.
Interactive demo
Interactive rendering on ZJU-MoCap
Please see INSTALL.md to download the dataset. We provide the pretrained models at here.
Take the rendering on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313/final.pth
. -
Interactive rendering demo:
python gui.py --config configs/zjumocap/dymap_313.py fast_render True
Interactive rendering on NHR
Please see INSTALL.md to download the dataset. We provide the pretrained models at here.
Take the rendering on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1/final.pth
. -
Interactive rendering demo:
python gui.py --config configs/nhr/sport1.py fast_render True
Run the code on ZJU-MoCap
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
Test on ZJU-MoCap
Take the test on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313/final.pth
. -
Test on unseen views:
python run.py --config configs/zjumocap/dymap_313.py mode evaluate fast_render True
Visualization on ZJU-MoCap
Take the visualization on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313
. -
Visualization:
- Visualize free-viewpoint videos
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fast_render True
- Visualize novel views of single frame
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_time True fast_render True
- Visualize the dynamic scene with fixed camera
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_view True fast_render True
- Visualize mesh
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_mesh True fast_render True
Training on ZJU-MoCap
Take the training on sequence 313
as an example.
- Train:
# training python train_net.py --config configs/zjumocap/dymap_313.py # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/zjumocap/dymap_313.py
- Post-process the trained model:
python run.py --config configs/zjumocap/dymap_313.py mode visualize occ_grid True
- Tensorboard:
tensorboard --logdir data/record/zjumocap
Run the code on NHR
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
Test on NHR
Take the test on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1/final.pth
. -
Test on unseen views:
python run.py --config configs/nhr/sport1.py mode evaluate fast_render True
Visualization on NHR
Take the visualization on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1
. -
Visualization:
- Visualize novel views
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fast_render True
- Visualize novel views of single frame
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_time True fast_render True
- Visualize the dynamic scene with fixed camera
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_view True fast_render True
- Visualize mesh
python run.py --config configs/nhr/sport1.py mode visualize vis_mesh True fast_render True
Training on NHR
Take the training on sequence sport1
as an example.
- Train:
# training python train_net.py --config configs/nhr/sport1.py # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/nhr/sport1.py
- Post-process the trained model:
python run.py --config configs/nhr/sport1.py mode visualize occ_grid True
- Tensorboard:
tensorboard --logdir data/record/nhr
Citation
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{peng2023representing,
title={Representing Volumetric Videos as Dynamic MLP Maps},
author={Peng, Sida and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
booktitle={CVPR},
year={2023}
}