SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

Zhaoxi Chen  Guangcong Wang  Ziwei Liu

S-Lab, Nanyang Technological University

TL;DR

SceneDreamer learns to generate unbounded 3D scenes from in-the-wild 2D image collections.
Our method can synthesize diverse landscapes across different styles, with 3D consistency, well-defined depth, and free camera trajectory.

Paper | Project Page | Video | Hugging Face 🤗

Updates

[04/2023] Hugging Face demo released!

[04/2023] Inference code released!

[02/2023] Paper uploaded to arXiv.

Citation

If you find our work useful for your research, please consider citing this paper:

@inproceedings{chen2023sd,
    title={SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections},
    author={Chen, Zhaoxi and Wang, Guangcong and Liu, Ziwei},
    booktitle={arXiv},
    year={2023},
}

Installation

We highly recommend using Anaconda to manage your python environment. You can setup the required environment by the following commands:

# install python dependencies
conda env create -f environment.yaml
conda activate scenedreamer

# compile third party libraries
export CUDA_VERSION=$(nvcc --version| grep -Po "(\d+\.)+\d+" | head -1)
CURRENT=$(pwd)
for p in correlation channelnorm resample2d bias_act upfirdn2d; do
    cd imaginaire/third_party/${p};
    rm -rf build dist *info;
    python setup.py install;
    cd ${CURRENT};
done

for p in gancraft/voxlib; do
  cd imaginaire/model_utils/${p};
  make all
  cd ${CURRENT};
done

cd gridencoder
python setup.py build_ext --inplace
python -m pip install .
cd ${CURRENT}

# Now, all done!

Inference

Download Pretrained Models

Please download our checkpoints from Google Drive to run the following inference scripts. You may store the checkpoint at the root directory of this repo:

├── ...
└── SceneDreamer
    ├── inference.py
    ├── README.md
    └── scenedreamer_released.pt

Render!

You can run the following command to generate your own 3D world!

python inference.py --config configs/scenedreamer_inference.yaml --output_dir ./test/ --seed 8888 --checkpoint ./scenedreamer_released.pt

The results will be saved under ./test as the following structures:

├── ...
└── test
    └── camera_{:02d} # camera mode for trajectory
        ├── rgb_render # per frame RGB renderings
            ├── 00000.png
            ├── 00001.png
            └── ...
        ├── rgb_render.mp4 # rendered video
        ├── height_map.png # height map
        ├── semantic_map.png # semantic map
        └── style.npy # sampled style code

Furthermore, you can modify the parameters for rendering in scenedreamer_inference.yaml, detailed as follows:

Here is a sampled scene with our default rendering parameters:

Gradio Demo

You can also locally launch our demo with gradio UI by:

python app_gradio.py

Alternatively, you can run the demo online

License

Distributed under the S-Lab License. See LICENSE for more information. Part of the codes are also subject to the LICENSE of imaginaire by NVIDIA.

Acknowledgements

This work is supported by the National Research Foundation, Singapore under its AI Singapore Programme, NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

SceneDreamer is implemented on top of the imaginaire. Thanks torch-ngp for the pytorch CUDA implementation of neural hash grid.