Versatile Diffusion
This repo hosts the official implementary of:
Xingqian Xu, Atlas Wang, Eric Zhang, Kai Wang, and Humphrey Shi, Versatile Diffusion: Text, Images and Variations All in One Diffusion Model, Paper arXiv Link.
News
- [2023.02.07]: Our new demo is up and running on ๐คHugging Face!
- [2023.02.07]: A major update on Versatile Diffusion's code in which model codes are better structured, and all applications are supported in a convenience WebUI.
- [Previous]: Please see the deprecated README.
Introduction
We built Versatile Diffusion (VD), the first unified multi-flow multimodal diffusion framework, as a step towards Universal Generative AI. Versatile Diffusion can natively support image-to-text, image-variation, text-to-image, and text-variation, and can be further extended to other applications such as semantic-style disentanglement, image-text dual-guided generation, latent image-to-text-to-image editing, and more. Future versions will support more modalities such as speech, music, video and 3D.
Network and Framework
One single flow of Versatile Diffusion contains a VAE, a diffuser, and a context encoder, and thus handles one task (e.g., text-to-image) under one data type (e.g., image) and one context type (e.g., text). The multi-flow structure of Versatile Diffusion shows in the following diagram:
According to Versatile Diffusion, we further proposed a generalized multi-flow multimodal framework with VAEs, context encoders, and diffusers containing three layers (i.e., global, data, and context layers). To involve a new multimodal task in this framework, we bring out the following requirements:
- The design of the core diffuser should contain shared global layers, swappable data, and context layers that will be correspondingly activated based on data and context types.
- The choice of VAEs should smoothly map data onto highly interpretable latent spaces.
- The choice of context encoders should jointly minimize the cross-modal statistical distance on all supported content types.
Performance
Data
We use Laion2B-en with customized data filters as our main dataset. Since Laion2B is very large and typical training is less than one epoch, we usually do not need to download the complete dataset for training. Same story for VDs.
Directory of Laion2B for our code:
โโโ data
โย ย โโโ laion2b
โย ย โโโ data
โ ย ย โโโ 00000.tar
โ ย ย โโโ 00000.parquet
โ ย ย โโโ 00000_stats.jsom_
โ ย ย โโโ 00001.tar
โ ย ย โโโ ...
These compressed data are generated with img2dataset API official github link.
Setup
conda create -n versatile-diffusion python=3.8
conda activate versatile-diffusion
conda install pytorch==1.12.1 torchvision=0.13.1 -c pytorch
[Alternatively] pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
pip install -r requirements.txt
Pretrained models
All pretrained models can be downloaded from Hugging Face link. The pretrained folder should include the following files:
โโโ pretrained
โย ย โโโ kl-f8.pth
โย ย โโโ optimus-vae.pth
โ โโโ vd-four-flow-v1-0.pth
โ โโโ vd-four-flow-v1-0-fp16.pth
Model named with -fp16 are models with float16 parameters, which is half size of the float32 models.
Inference
We now provide a convenience WebUI app.py
that supports all applications. Start the WebUI with the following command:
python app.py
The WebUI contains the following new features:
- Improved and enhanced Image-Variation
- New Triple-Context Image Blender (support two images with optional masks + text)
- New Multi-Context Image Blender (support up to four images with optional masks + text)
The following old features have been temporarily disabled:
- I2T2I is offline. We are seeking a better way of image editing.
Gallery
Text-to-Image
Image-Variation
Image-Variation with Semantic Focus
Dual-guided
Citation
@article{xu2022versatile,
title = {Versatile Diffusion: Text, Images and Variations All in One Diffusion Model},
author = {Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, Humphrey Shi},
year = 2022,
url = {https://arxiv.org/abs/2211.08332},
eprint = {2211.08332},
archiveprefix = {arXiv},
primaryclass = {cs.CV}
}
Acknowledgement
Part of the codes reorganizes/reimplements code from the following repositories: LDM official Github, which also oriented from DDPM official Github.