thu-ml/CRM thu-ml/CRM

  • Stars
    star
    520
  • Rank 85,117 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created 8 months ago
  • Updated 8 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
thu-ml/CRM
github

thu-ml

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[ECCV 2024] Single Image to 3D Textured Mesh in 10 seconds with Convolutional Reconstruction Model.

Convolutional Reconstruction Model

Official implementation for CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model.

CRM is a feed-forward model which can generate 3D textured mesh in 10 seconds.

Project Page | Arxiv | HF-Demo | Weights

teaser.mp4

Try CRM 🍻

Install

Step 1 - Base

Install package one by one, we use python 3.9

pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117
pip install torch-scatter==2.1.1 -f https://data.pyg.org/whl/torch-1.13.1+cu117.html
pip install kaolin==0.14.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.13.1_cu117.html
pip install -r requirements.txt

besides, one by one need to install xformers manually according to the official doc (conda no need), e.g.

pip install ninja
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers

Step 2 - Nvdiffrast

Install nvdiffrast according to the official doc, e.g.

pip install git+https://github.com/NVlabs/nvdiffrast

Inference

We suggest gradio for a visualized inference.

gradio app.py

image

For inference in command lines, simply run

CUDA_VISIBLE_DEVICES="0" python run.py --inputdir "examples/kunkun.webp"

It will output the preprocessed image, generated 6-view images and CCMs and a 3D model in obj format.

Tips: (1) If the result is unsatisfatory, please check whether the input image is correctly pre-processed into a grey background. Otherwise the results will be unpredictable. (2) Different from the Huggingface Demo, this official implementation uses UV texture instead of vertex color. It has better texture than the online demo but longer generating time owing to the UV texturing.

Todo List

  • Release inference code.
  • Release pretrained models.
  • Optimize inference code to fit in low memery GPU.
  • Upload training code.

Acknowledgement

Citation

@article{wang2024crm,
  title={CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model},
  author={Zhengyi Wang and Yikai Wang and Yifei Chen and Chendong Xiang and Shuo Chen and Dajiang Yu and Chongxuan Li and Hang Su and Jun Zhu},
  journal={arXiv preprint arXiv:2403.05034},
  year={2024}
}

More Repositories

1

tianshou

An elegant PyTorch deep reinforcement learning library.
Python
7,810
star
2

zhusuan

A probabilistic programming library for Bayesian deep learning, generative models, based on Tensorflow
Python
2,202
star
3

prolificdreamer

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (NeurIPS 2023 Spotlight)
Python
1,472
star
4

unidiffuser

Code and models for the paper "One Transformer Fits All Distributions in Multi-Modal Diffusion"
Python
1,365
star
5

ares

A Python library for adversarial machine learning focusing on benchmarking adversarial robustness.
Python
480
star
6

SageAttention

Quantized Attention that achieves speedups of 2.1x and 2.7x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.
Python
222
star
7

controlvideo

Official implementation for "ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing"
Python
181
star
8

warplda

Cache efficient implementation for Latent Dirichlet Allocation
C++
161
star
9

3D_Corruptions_AD

Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving, CVPR 2023
Python
119
star
10

low-bit-optimizers

Low-bit optimizers for PyTorch
Python
114
star
11

MMTrustEval

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)
Python
89
star
12

stochastic_gcn

Stochastic training of graph convolutional networks
Python
84
star
13

RoboticsDiffusionTransformer

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
Python
72
star
14

Attack-Bard

Python
53
star
15

DPM-Solver-v3

Official code for "DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics" (NeurIPS 2023)
48
star
16

tianshou-docs-zh_CN

ε€©ζŽˆδΈ­ζ–‡ζ–‡ζ‘£
TeX
46
star
17

Prior-Guided-RGF

Python
41
star
18

zh-clip

Python
41
star
19

SRPO

Codes accompanying the paper "Score Regularized Policy Optimization through Diffusion Behavior" (ICLR 2024).
Python
36
star
20

vflow

Official code for "VFlow: More Expressive Generative Flows with Variational Data Augmentation" (ICML 2020)
Python
35
star
21

AT3D

Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition, CVPR 2023, Highlight
Python
34
star
22

implicit-normalizing-flows

Code for "Implicit Normalizing Flows" (ICLR 2021 spotlight)
Python
34
star
23

HiDe-Prompt

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality (NeurIPS 2023, Spotlight)
Python
30
star
24

BigTopicModel

Big Topic Model is a fast engine for running large-scale Topic Models.
C++
22
star
25

NUNO

[ICML 2023] Non-Uniform Neural Operator (NUNO)
Python
18
star
26

IODF

C++
16
star
27

fpovi

Code for "Function Space Particle Optimization for Bayesian Neural Networks"
Python
16
star
28

CF-UIcA

Code for "Collaborative Filtering with User-Item Co-Autoregressive Models"
Python
15
star
29

Zhusuan-Jittor

Zhusuan with backend Jittor
Python
14
star
30

LM-Calibration

Python
12
star
31

mmdcgm-ssl

mmDCGMs for accurate classification and excellent class-conditional generation in semi-supervised learning
Python
11
star
32

Zhusuan-PaddlePaddle

Zhusuan with backend PaddlePaddle
Python
8
star
33

ood-dgm

Python
8
star
34

MEM_DGM

Code for "Learning to Generate with Memory"
Python
8
star
35

ProbML-book-solution

Jupyter Notebook
7
star
36

adversarial_training_imagenet

Python
7
star
37

pmd

Population matching discrepancy
Python
7
star
38

CEURL

Official implementation for "PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning" (NeurIPS 2024)
Python
7
star
39

VCAS

Official code for "Efficient Backpropagation with Variance Controlled Adaptive Sampling" (ICLR 2024)
Python
6
star
40

imagenet-a-plus

Python
4
star
41

wmvl

Code for "A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models"
Jupyter Notebook
3
star
42

sEM-vr

code for pLSA and LDA in the paper "Stochastic Expectation Maximization with Variance Reduction"
C++
3
star
43

Efficient-Diffusion-Alignment

Official Codebase for "Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control" (NeurIPS 2024)
3
star
44

Noise-Contrastive-Alignment

Code accompanying the paper "Noise Contrastive Alignment of Language Models with Explicit Rewards"
2
star
45

i-DODE

Official code for "Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs" (ICML 2023)
1
star
46

CCA

Codes accompanying the paper "Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment"
Python
1
star
47

ACTNN-PaddlePaddle

Python
1
star
48

DBIM

Official codebase for "Diffusion Bridge Implicit Models" (https://arxiv.org/abs/2405.15885).
1
star
49

Jetfire-INT8Training

Jupyter Notebook
1
star