Jiawei Ren* Liang Pan* Jiaxiang Tang Chi Zhang Ang Cao Gang Zeng Ziwei Liuâ€
S-Lab, Nanyang Technological University 
Shanghai AI Laboratory 
Peking University   University of Michigan  
*equal contribution
†corresponding author
Peking University   University of Michigan  
*equal contribution
†corresponding author
Arxiv 2023
examples.mp4
# install customized diffusers
pip install ./diffusers
pip install -r requirements.txt
# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization
# simple-knn
pip install ./simple-knn
# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/
# kiuikit
pip install git+https://github.com/ashawkey/kiuikit
Tested on:
- python 3.8 & torch 2.1 & CUDA 11.8 on an 80GB A100.
- Stage I: 2min. Stage II: 4.5min. Stage III: 3.5min.
- python 3.8 & torch 2.0 & CUDA 11.6 on an 24GB A5000.
- Stage I: 4.5min. Stage II: 9min. Stage III: 5.5min with
oom_hack.
- Stage I: 4.5min. Stage II: 9min. Stage III: 5.5min with
# generate driving video
python gen_vid.py --name anya_rgba --seed 42 --bg white
# Stage I: train 500 iters (~2min) and export ckpt & coarse_mesh to logs
python main.py --config configs/image.yaml input=data/anya_rgba.png save_path=anya
# Stage II: temporal optimization stage (export meshes by default)
python main_4d.py --config configs/4d.yaml input=data/anya_rgba.png save_path=anya
# Stage III: texture optimization (optional, it requires large GPU memory and we are optimzing it)
python main2_4d.py --config configs/4d_svd.yaml input=data/anya_rgba.png save_path=anya
# to turn on viser GUI, add `gui=True`, e.g.:
python main.py --config configs/image.yaml input=data/anya_rgba.png save_path=anya gui=TrueMeshes will be automatically exported to logs in Stage II. Visulizations will be saved to vis_data.
- Install the Stop-motion-OBJ add-on
- File -> Import -> Mesh Sequence
- Go to
logsdirectory, type in the file name (e.g., 'anya'), and tickMaterial per Frame.
blender_lowfps.mov
-
CUDA OOM.
- Stage I: Reduce batch_size to 4.
- Stage II: Reduce n_views to 1.
- Stage III: Add argument
oom_hack=Trueor disable SVD by setting lambda_svd to 0.
-
Black video after running
gen_vid.py.- Make sure pytorch version is >=2.0
This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!
@article{ren2023dreamgaussian4d,
title={DreamGaussian4D: Generative 4D Gaussian Splatting},
author={Ren, Jiawei and Pan, Liang and Tang, Jiaxiang and Zhang, Chi and Cao, Ang and Zeng, Gang and Liu, Ziwei},
journal={arXiv preprint arXiv:2312.17142},
year={2023}
}