thu-ml/controlvideo thu-ml/controlvideo

  • Stars
    star
    181
  • Rank 212,110 (Top 5 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 1 year ago
  • Updated over 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
thu-ml/controlvideo
github

thu-ml

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Official implementation for "ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing"

ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing

This is the official implementation for "ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing". The project page is available here. Code will be released soon.

Overview

ControlVideo incorporates visual conditions for all frames to amplify the source video's guidance, key-frame attention that aligns all frames with a selected one and temporal attention modules succeeded by a zero convolutional layer for temporal consistency and faithfulness. The three key components and corresponding fine-tuned parameters are designed by a systematic empirical study. Built upon the trained ControlVideo, during inference, we employ DDIM inversion and then generate the edited video using the target prompt via DDIM sampling. image

Main Results

image

To Do List

  • Multi Controls Code Organization
  • Support ControlNet 1.1
  • Support Attention Control
  • More Applications such as Image-Guided Video Generation
  • Hugging Face
  • More Sampler

Environment

conda env create -f environment.yml

The environment is similar to Tune-A-Video

Prepare Pretrained Text-to-Image Diffusion Model

Download the Stable Diffusion 1.5 and ControlNet 1.0 for canny, HED, depth and pose. Put them in ./ .

Quick Start

python main.py --control_type hed --video_path videos/car10.mp4 --source 'a car' --target 'a red car' --out_root outputs/ --max_step 300

The control_type is the type of controls, which is chosen from canny/hed/depth/pose. The video_path is the path to the input video. The source is the source prompt for the source video. The target is the target prompt. The max_step is the step for training. The out_root is the path for saving results.

Run More Demos

Download the data and put them in videos/.

python run_demos.py

References

If you find this repository helpful, please cite as:

@article{zhao2023controlvideo,
  title={ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing},
  author={Zhao, Min and Wang, Rongzhen and Bao, Fan and Li, Chongxuan and Zhu, Jun},
  journal={arXiv preprint arXiv:2305.17098},
  year={2023}
}

This implementation is based on Tune-A-Video and Video-p2p.

More Repositories

1

tianshou

An elegant PyTorch deep reinforcement learning library.
Python
7,810
star
2

zhusuan

A probabilistic programming library for Bayesian deep learning, generative models, based on Tensorflow
Python
2,202
star
3

prolificdreamer

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (NeurIPS 2023 Spotlight)
Python
1,472
star
4

unidiffuser

Code and models for the paper "One Transformer Fits All Distributions in Multi-Modal Diffusion"
Python
1,365
star
5

CRM

[ECCV 2024] Single Image to 3D Textured Mesh in 10 seconds with Convolutional Reconstruction Model.
Python
520
star
6

ares

A Python library for adversarial machine learning focusing on benchmarking adversarial robustness.
Python
480
star
7

SageAttention

Quantized Attention that achieves speedups of 2.1x and 2.7x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.
Python
222
star
8

warplda

Cache efficient implementation for Latent Dirichlet Allocation
C++
161
star
9

3D_Corruptions_AD

Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving, CVPR 2023
Python
119
star
10

low-bit-optimizers

Low-bit optimizers for PyTorch
Python
114
star
11

MMTrustEval

A toolbox for benchmarking trustworthiness of multimodal large language models (MultiTrust, NeurIPS 2024 Track Datasets and Benchmarks)
Python
89
star
12

stochastic_gcn

Stochastic training of graph convolutional networks
Python
84
star
13

RoboticsDiffusionTransformer

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
Python
72
star
14

Attack-Bard

Python
53
star
15

DPM-Solver-v3

Official code for "DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics" (NeurIPS 2023)
48
star
16

tianshou-docs-zh_CN

ε€©ζŽˆδΈ­ζ–‡ζ–‡ζ‘£
TeX
46
star
17

Prior-Guided-RGF

Python
41
star
18

zh-clip

Python
41
star
19

SRPO

Codes accompanying the paper "Score Regularized Policy Optimization through Diffusion Behavior" (ICLR 2024).
Python
36
star
20

vflow

Official code for "VFlow: More Expressive Generative Flows with Variational Data Augmentation" (ICML 2020)
Python
35
star
21

AT3D

Towards Effective Adversarial Textured 3D Meshes on Physical Face Recognition, CVPR 2023, Highlight
Python
34
star
22

implicit-normalizing-flows

Code for "Implicit Normalizing Flows" (ICLR 2021 spotlight)
Python
34
star
23

HiDe-Prompt

Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality (NeurIPS 2023, Spotlight)
Python
30
star
24

BigTopicModel

Big Topic Model is a fast engine for running large-scale Topic Models.
C++
22
star
25

NUNO

[ICML 2023] Non-Uniform Neural Operator (NUNO)
Python
18
star
26

IODF

C++
16
star
27

fpovi

Code for "Function Space Particle Optimization for Bayesian Neural Networks"
Python
16
star
28

CF-UIcA

Code for "Collaborative Filtering with User-Item Co-Autoregressive Models"
Python
15
star
29

Zhusuan-Jittor

Zhusuan with backend Jittor
Python
14
star
30

LM-Calibration

Python
12
star
31

mmdcgm-ssl

mmDCGMs for accurate classification and excellent class-conditional generation in semi-supervised learning
Python
11
star
32

Zhusuan-PaddlePaddle

Zhusuan with backend PaddlePaddle
Python
8
star
33

ood-dgm

Python
8
star
34

MEM_DGM

Code for "Learning to Generate with Memory"
Python
8
star
35

ProbML-book-solution

Jupyter Notebook
7
star
36

adversarial_training_imagenet

Python
7
star
37

pmd

Population matching discrepancy
Python
7
star
38

CEURL

Official implementation for "PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning" (NeurIPS 2024)
Python
7
star
39

VCAS

Official code for "Efficient Backpropagation with Variance Controlled Adaptive Sampling" (ICLR 2024)
Python
6
star
40

imagenet-a-plus

Python
4
star
41

wmvl

Code for "A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models"
Jupyter Notebook
3
star
42

sEM-vr

code for pLSA and LDA in the paper "Stochastic Expectation Maximization with Variance Reduction"
C++
3
star
43

Efficient-Diffusion-Alignment

Official Codebase for "Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control" (NeurIPS 2024)
3
star
44

Noise-Contrastive-Alignment

Code accompanying the paper "Noise Contrastive Alignment of Language Models with Explicit Rewards"
2
star
45

i-DODE

Official code for "Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs" (ICML 2023)
1
star
46

CCA

Codes accompanying the paper "Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment"
Python
1
star
47

ACTNN-PaddlePaddle

Python
1
star
48

DBIM

Official codebase for "Diffusion Bridge Implicit Models" (https://arxiv.org/abs/2405.15885).
1
star
49

Jetfire-INT8Training

Jupyter Notebook
1
star