• Stars
    star
    164
  • Rank 230,032 (Top 5 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created 12 months ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

KandinskyVideo — multilingual end-to-end text2video latent diffusion model

Kandinsky Video — a new text-to-video generation model

SoTA quality among open-source solutions

This repository is the official implementation of Kandinsky Video model.

Paper | Project | Hugging Face Spaces | Telegram-bot | Habr post | Our text-to-image model | Replicate


Kandinsky Video is a text-to-video generation model, which is based on the FusionFrames architecture and Kandinsky 3.0 text-to-image model, consisting of two main stages: keyframe generation and interpolation. Our approach for temporal conditioning allows us to generate videos with high-quality appearance, smoothness and dynamics.

Pipeline


The encoded text prompt enters the U-Net keyframe generation model with temporal layers or blocks, and then the sampled latent keyframes are sent to the latent interpolation model in such a way as to predict three interpolation frames between two keyframes. A temporal MoVQ-GAN decoder is used to get the final video result.

Architecture details

  • Text encoder (Flan-UL2) - 8.6B
  • Latent Diffusion U-Net3D - 4.0B
  • MoVQ encoder/decoder - 256M

How to use

Check our jupyter notebooks with examples in ./examples folder

1. text2video

from video_kandinsky3 import get_T2V_pipeline

t2v_pipe = get_T2V_pipeline('cuda', fp16=True)

pfps = 'medium' # ['low', 'medium', 'high']
video = t2v_pipe(
    'a red car is drifting on the mountain road, close view, fast movement',
    width=640, height=384, fps=fps
)

Results

"A car moving on the road from the sea to the mountains" "A red car drifting, 4k video" "Chemistry laboratory, chemical explosion, 4k" "Erupting volcano raw power, molten lava, and the forces of the Earth"
"Luminescent jellyfish swims underwater, neon, 4k" "Majestic waterfalls in a lush rainforest power, mist, and biodiversity" "White ghost flies through a night clearing, 4k" "Wildlife migration herds on the move, crossing landscapes in harmony"
"Majestic humpback whale breaching power, grace, and ocean spectacle" "Evoke the sense of wonder in a time-lapse journey through changing seasons" "Explore the fascinating world of underwater creatures in a visually stunning sequence" "Polar ice caps the pristine wilderness of the Arctic and Antarctic"
"Rolling waves on a sandy beach relaxation, rhythm, and coastal beauty" "Sloth in slow motion deliberate movements, relaxation, and arboreal life" "Time-lapse of a flower blooming growth, beauty, and the passage of time" "Craft a heartwarming narrative showcasing the bond between a human and their loyal pet companion"

Authors

BibTeX

If you use our work in your research, please cite our publication:

@article{arkhipkin2023fusionframes,
  title     = {FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipeline},
  author    = {Arkhipkin, Vladimir and Shaheen, Zein and Vasilev, Viacheslav and Dakhova, Elizaveta and Kuznetsov, Andrey and Dimitrov, Denis},
  journal   = {arXiv preprint arXiv:2311.13073},
  year      = {2023}, 
}

More Repositories

1

Kandinsky-2

Kandinsky 2 — multilingual text2image latent diffusion model
Jupyter Notebook
2,745
star
2

ru-gpts

Russian GPT3 models.
Python
2,077
star
3

ru-dalle

Generate images from texts. In Russian
Jupyter Notebook
1,640
star
4

ghost

A new one shot face swap approach for image and video domains
Python
1,190
star
5

ner-bert

BERT-NER (nert-bert) with google bert https://github.com/google-research.
Jupyter Notebook
405
star
6

ru-dolph

RUDOLPH: One Hyper-Tasking Transformer can be creative as DALL-E and GPT-3 and smart as CLIP
Jupyter Notebook
242
star
7

Real-ESRGAN

PyTorch implementation of Real-ESRGAN model
Python
201
star
8

mgpt

Multilingual Generative Pretrained Model
Jupyter Notebook
196
star
9

ru-clip

CLIP implementation for Russian language
Jupyter Notebook
134
star
10

sage

SAGE: Spelling correction, corruption and evaluation for multiple languages
Jupyter Notebook
129
star
11

ruGPT3_demos

121
star
12

deforum-kandinsky

Kandinsky x Deforum — generating short animations
Python
102
star
13

digital_peter_aij2020

Materials of the AI Journey 2020 competition dedicated to the recognition of Peter the Great's manuscripts, https://ai-journey.ru/contest/task01
Jupyter Notebook
66
star
14

music-composer

Python
62
star
15

ru-prompts

Python
57
star
16

gigachat

Библиотека для доступа к GigaChat
Python
57
star
17

MERA

MERA (Multimodal Evaluation for Russian-language Architectures) is a new open benchmark for the Russian language for evaluating fundamental models.
Jupyter Notebook
55
star
18

fusion_brain_aij2021

Creating multimodal multitask models
Jupyter Notebook
50
star
19

augmentex

Augmentex — a library for augmenting texts with errors
Python
48
star
20

model-zoo

NLP model zoo for Russian
45
star
21

OCR-model

An easy-to-run OCR model pipeline based on CRNN and CTC loss
Python
43
star
22

StackMix-OCR

Jupyter Notebook
40
star
23

MoVQGAN

MoVQGAN - model for the image encoding and reconstruction
Jupyter Notebook
35
star
24

tuned-vq-gan

Jupyter Notebook
28
star
25

ReadingPipeline

Text reading pipeline that combines segmentation and OCR-models.
Python
26
star
26

DataProcessingFramework

Framework for processing and filtering datasets
Python
25
star
27

htr_datasets

Repository containing our datasets for HTR (handwritten text recognition) task.
Jupyter Notebook
23
star
28

CerberusDet

CerberusDet: Unified Multi-Task Object Detection
Python
23
star
29

fbc3_aij2023

Jupyter Notebook
21
star
30

mineral-recognition

Python
20
star
31

DigiTeller

18
star
32

fbc2_aij2022

FusionBrain Challenge 2.0: creating multimodal multitask model
Python
16
star
33

combined_solution_aij2019

AI Journey 2019: Combined Solution
Python
15
star
34

SEGM-model

An easy-to-run semantic segmentation model based on Unet
Python
13
star
35

railway_infrastructure_detection_aij2021

AI Journey Contest 2021: AITrain
Python
13
star
36

no_fire_with_ai_aij2021

AI Journey Contest 2021: NoFireWithAI
Jupyter Notebook
13
star
37

ControlledNST

An implementation of Neural Style Transfer in PyTorch.
Jupyter Notebook
8
star
38

kandinsky3-diffusers

Python
6
star
39

mchs-wildfire

Соревнование по классификации лесных пожаров
Jupyter Notebook
4
star
40

no_flood_with_ai_aij2020

Материалы соревнования AI Journey 2020, посвященного прогнозированию паводков на реке Амур, https://ai-journey.ru/contest/task02
Jupyter Notebook
4
star
41

Zoom_In_Video_Kandinsky

Framework for creating Zoom in / Zoom out video based on inpainting Kandinsky
Jupyter Notebook
2
star
42

langchain-gigachat

Python
1
star
43

paper_persi_chat

PaperPersiChat: Scientific Paper Discussion Chatbot using Transformers and Discourse Flow Management
Jupyter Notebook
1
star