• Stars
    star
    528
  • Rank 83,327 (Top 2 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 1 year ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information

ControlLoRA: A Light Neural Network To Control Stable Diffusion Spatial Information

EN | 中文

By combining the ideas of lllyasviel/ControlNet and cloneofsimo/lora, we can easily fine-tune stable diffusion to achieve the purpose of controlling its spatial information, with ControlLoRA, a simple and small (~7M parameters, ~25M storage space) network.

ControlNet is large and it's not easy to send to your friends. With the idea of LoRA, we don't even need to transfer the entire stable diffusion model. Use the 25M ControlLoRA to save your time.

You could use gradio apps in the apps directory to try the pretrained models. More dataset types of models and their supporting gradio apps wanted. The annotator directory is borrowed from ControlNet.

You could download some pretrained models from huggingface. Note that I only used 100 MPII pictures for the training of the openpose, so the model effect is not good. So I suggest you train your own ControlLoRA.

Features & News

2023/02/22 - Add ControlLora v2, which decomposites the prompt features and the spatial information with smaller size (~5M parameters, ~20M storage space). You could do something like: training on sd v1.5 then inference on anything v3.0 .

How To Train

Refer to the script in the tasks directory. I highly refer to the training code from diffusers.

You could add or modify config file in the configs directory to custom the ControlLoRA model architecture. To enhance the effect of the model, you could change some blocks to other residual block types of diffusers and you could increase the number of layers of blocks by modify the config files.

Work In Progress

  • More type tasks mentioned in ControlNet.

  • Experiment of mixing LoRA and ControlLoRA.

    We could inject pretrained LoRA models before the ControlLoRA. See mix_lora_and_control_lora.py for more details.

    p portrait of male HighCWu

ControlLoRA with Canny Edge

sd-diffusiondb-canny-model-control-lora, on 100 openpose pictures, 30k training steps

Stable Diffusion 1.5 + ControlLoRA (using simple Canny edge detection)

python apps/gradio_canny2image.py

Highly referred to the ControlNet codes.

The Gradio app also allows you to change the Canny edge thresholds. Just try it for more details.

Prompt: "bird" p

Prompt: "cute dog" p

ControlLoRA with Human Pose

sd-mpii-pose-model-control-lora, on 100 openpose pictures, 30k training steps

Stable Diffusion 1.5 + ControlLoRA (using human pose)

python apps/gradio_pose2image.py

Highly referred to the ControlNet codes.

Apparently, this model deserves a better UI to directly manipulate pose skeleton. However, again, Gradio is somewhat difficult to customize. Right now you need to input an image and then the Openpose will detect the pose for you.

Note that I only used 100 MPII pictures for the training of the openpose, so the model effect is not good. So I suggest you train your own ControlLoRA.

Prompt: "Chief in the kitchen" p

Prompt: "An astronaut on the moon" p

PS: I don't know why my gallery didn't show the full images and I should click an output to show the full result of one of the outputs, like this: p

Discuss together

QQ Group: 艾梦的小群

QQ Channel: 艾梦的AI造梦堂

Discord: AI Players - AI Dream Bakery

Citation

@software{wu2023controllora,
    author = {Wu Hecong},
    month = {2},
    title = {{ControlLoRA: A Light Neural Network To Control Stable Diffusion Spatial Information}},
    url = {https://github.com/HighCWu/ControlLoRA},
    version = {1.0.0},
    year = {2023}
}

More Repositories

1

anime_biggan_toy

Generate Amazing Anime Pictures With BigGAN. Just Have Fun !!!
Jupyter Notebook
143
star
2

control-lora-v2

ControlLoRA Version 2: A Lightweight Neural Network To Control Stable Diffusion Spatial Information Version 2
Python
80
star
3

waifu2x-tfjs

Image super-resolution using Tensorflow.js .
TypeScript
51
star
4

pixel-guide-diffusion-for-anime-colorization

Use denoising diffusion probabilistic model to do the anime colorization task.
Python
44
star
5

EasyOCR

Java OCR 识别组件(历史找回版本,基于Tesseract OCR 引擎)。能自动完成图片清理、识别 CAPTCHA 验证码图片内容的一体化工作。Java Image cleanup, OCR recognition component (based Tesseract OCR engine, automatically cleanup image and identification CAPTCHA verification code picture content).
38
star
6

stylegan2-faceapp

Try to use stylegan2 to achieve some facial special effects, still under construction
Python
26
star
7

keras-bert-tpu

Implementation of BERT that could load official pre-trained models for feature extraction and prediction on TPU
Python
17
star
8

face_morpher_plus

Face morpher plus based on facemorpher
Python
12
star
9

starganv2vc-paddle

Python
11
star
10

rwkv-paddle

Python
11
star
11

denoising-diffusion-paddle

Implementation of Denoising Diffusion Probabilistic Models in PaddlePaddle
Python
10
star
12

DECA-Paddle-Inference

Python
6
star
13

DiffSinger-ONNX-Infer

DiffSinger ONNX Inference
Jupyter Notebook
6
star
14

SelfGAN

My pytorch implementation of 'SelfGAN--Not A GAN But Punch Itself'.
Jupyter Notebook
5
star
15

control-lora-v3

ControlLoRA Version 3: LoRA Is All You Need to Control the Spatial Information of Stable Diffusion.
Python
4
star
16

stylegan2-paddle

Simplest working implementation of Stylegan2 in PaddlePaddle. Enabling everyone to experience disentanglement
Python
4
star
17

beat-interpolator

Interpolate the latents of your DL model to follow the beat of the music
Python
3
star