Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation

⚡️News

❗️2023.04.10: We've opened the HuggingFace Demo! Also, we fixed minor issues, including the seed not being fixed.

❗️2023.03.31: We found that we typed an incorrect version of the model for point cloud inference. The fixed commit produces much better results.

Introduction

We introduce 3DFuse, a novel framework that incorporates 3D awareness into pretrained 2D diffusion models, enhancing the robustness and 3D consistency of score distillation-based methods. For more details, please visit our project page!

🔥TODO

• 3D Generation/Gradio Demo Code
• HuggingFace🤗 Demo Release
• Colab Demo Release
• Mesh Converting Code

Installation

Please follow installation.

Interactive Gradio App

for Text-to-3D / Image-to-3D

Enter your own prompt and enjoy! With this gradio app, you can preview the point cloud before 3D generation and determine the desired shape.

python gradio_app.py
# or python gradio_app.py --share

Text-to-3D Generation

After modifying the run.sh file with the desired prompt and hyperparameters, please execute the following command:

sh run.sh

Acknowledgement

We would like to acknowledge the contributions of public projects, including SJC and ControlNet whose code has been utilized in this repository.

Citation

@article{seo2023let,
  title={Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation},
  author={Seo, Junyoung and Jang, Wooseok and Kwak, Min-Seop and Ko, Jaehoon and Kim, Hyeonsu and Kim, Junho and Kim, Jin-Hwa and Lee, Jiyoung and Kim, Seungryong},
  journal={arXiv preprint arXiv:2303.07937},
  year={2023}
}