StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter
GongyeLiu, Menghan Xia*, Yong Zhang, Haoxin Chen, Jinbo Xing,
Xintao Wang, Yujiu Yang*, Ying Shan
(* corresponding authors)
From Tsinghua University and Tencent AI Lab.
π Introduction
TL;DR: We propose StyleCrafter, a generic method that enhances pre-trained T2V models with style control, supporting Style-Guided Text-to-Image Generation and Style-Guided Text-to-Video Generation.
1. ββ Style-Guided Text-to-Video Generation.
2. Style-Guided Text-to-Image Generation.
π Changelog
- [2023.12.08]: π₯π₯ Release the Huggingface online demo.
- [2023.12.05]: π₯π₯ Release the code and checkpoint.
- [2023.11.30]: π₯π₯ Release the project page.
β³ TODO
- Remove Video Watermark(due to trained on WebVid10M).
π§° Models
Model | Resolution | Checkpoint |
---|---|---|
StyleCrafter | 320x512 | Hugging Face |
It takes approximately 5 seconds to generate a 512Γ512 image and 85 seconds to generate a 320Γ512 video with 16 frames using a single NVIDIA A100 (40G) GPU. A GPU with at least 16G GPU memory is required to perform the inference process.
βοΈ Setup
conda create -n stylecrafter python=3.8.5
conda activate stylecrafter
pip install -r requirements.txt
π« Inference
- Download all checkpoints according to the instructions
- Run the commands in terminal.
# style-guided text-to-image generation
sh scripts/run_infer_image.sh
# style-guided text-to-video generation
sh scripts/run_infer_video.sh
- (Optional) Infernce on your own data according to the instructions
π¨βπ©βπ§βπ¦ Crafter Family
VideoCrafter1: Framework for high-quality text-to-video generation.
ScaleCrafter: Tuning-free method for high-resolution image/video generation.
TaleCrafter: An interactive story visualization tool that supports multiple characters.
LongerCrafter: Tuning-free method for longer high-quality video generation.
DynamiCrafter Animate open-domain still images to high-quality videos.
π’ Disclaimer
We develop this repository for RESEARCH purposes, so it can only be used for personal/research/non-commercial purposes.
π Acknowledgements
We would like to thank AK(@_akhaliq) for the help of setting up online demo.
π Contact
If your have any comments or questions, feel free to contact [email protected]