fcakyon/video-transformers fcakyon/video-transformers

  • Stars
    star
    116
  • Rank 294,156 (Top 6 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 2 years ago
  • Updated about 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
fcakyon/video-transformers
github

fcakyon

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Easiest way of fine-tuning HuggingFace video classification models

Easiest way of fine-tuning HuggingFace video classification models.

pypi version total downloads fcakyon twitter

πŸš€ Features

video-transformers uses:

and supports:

🏁 Installation

  • Install Pytorch:
conda install pytorch=1.11.0 torchvision=0.12.0 cudatoolkit=11.3 -c pytorch
  • Install pytorchvideo and transformers from main branch:
pip install git+https://github.com/facebookresearch/pytorchvideo.git
pip install git+https://github.com/huggingface/transformers.git
  • Install video-transformers:
pip install video-transformers

πŸ”₯ Usage

  • Prepare video classification dataset in such folder structure (.avi and .mp4 extensions are supported):
train_root
    label_1
        video_1
        video_2
        ...
    label_2
        video_1
        video_2
        ...
    ...
val_root
    label_1
        video_1
        video_2
        ...
    label_2
        video_1
        video_2
        ...
    ...
  • Fine-tune Timesformer (from HuggingFace) video classifier:
from torch.optim import AdamW
from video_transformers import VideoModel
from video_transformers.backbones.transformers import TransformersBackbone
from video_transformers.data import VideoDataModule
from video_transformers.heads import LinearHead
from video_transformers.trainer import trainer_factory
from video_transformers.utils.file import download_ucf6

backbone = TransformersBackbone("facebook/timesformer-base-finetuned-k400", num_unfrozen_stages=1)

download_ucf6("./")
datamodule = VideoDataModule(
    train_root="ucf6/train",
    val_root="ucf6/val",
    batch_size=4,
    num_workers=4,
    num_timesteps=8,
    preprocess_input_size=224,
    preprocess_clip_duration=1,
    preprocess_means=backbone.mean,
    preprocess_stds=backbone.std,
    preprocess_min_short_side=256,
    preprocess_max_short_side=320,
    preprocess_horizontal_flip_p=0.5,
)

head = LinearHead(hidden_size=backbone.num_features, num_classes=datamodule.num_classes)
model = VideoModel(backbone, head)

optimizer = AdamW(model.parameters(), lr=1e-4)

Trainer = trainer_factory("single_label_classification")
trainer = Trainer(datamodule, model, optimizer=optimizer, max_epochs=8)

trainer.fit()
  • Fine-tune ConvNeXT (from HuggingFace) + Transformer based video classifier:
from torch.optim import AdamW
from video_transformers import TimeDistributed, VideoModel
from video_transformers.backbones.transformers import TransformersBackbone
from video_transformers.data import VideoDataModule
from video_transformers.heads import LinearHead
from video_transformers.necks import TransformerNeck
from video_transformers.trainer import trainer_factory
from video_transformers.utils.file import download_ucf6

backbone = TimeDistributed(TransformersBackbone("facebook/convnext-small-224", num_unfrozen_stages=1))
neck = TransformerNeck(
    num_features=backbone.num_features,
    num_timesteps=8,
    transformer_enc_num_heads=4,
    transformer_enc_num_layers=2,
    dropout_p=0.1,
)

download_ucf6("./")
datamodule = VideoDataModule(
    train_root="ucf6/train",
    val_root="ucf6/val",
    batch_size=4,
    num_workers=4,
    num_timesteps=8,
    preprocess_input_size=224,
    preprocess_clip_duration=1,
    preprocess_means=backbone.mean,
    preprocess_stds=backbone.std,
    preprocess_min_short_side=256,
    preprocess_max_short_side=320,
    preprocess_horizontal_flip_p=0.5,
)

head = LinearHead(hidden_size=neck.num_features, num_classes=datamodule.num_classes)
model = VideoModel(backbone, head, neck)

optimizer = AdamW(model.parameters(), lr=1e-4)

Trainer = trainer_factory("single_label_classification")
trainer = Trainer(
    datamodule,
    model,
    optimizer=optimizer,
    max_epochs=8
)

trainer.fit()
  • Fine-tune Resnet18 (from HuggingFace) + GRU based video classifier:
from video_transformers import TimeDistributed, VideoModel
from video_transformers.backbones.transformers import TransformersBackbone
from video_transformers.data import VideoDataModule
from video_transformers.heads import LinearHead
from video_transformers.necks import GRUNeck
from video_transformers.trainer import trainer_factory
from video_transformers.utils.file import download_ucf6

backbone = TimeDistributed(TransformersBackbone("microsoft/resnet-18", num_unfrozen_stages=1))
neck = GRUNeck(num_features=backbone.num_features, hidden_size=128, num_layers=2, return_last=True)

download_ucf6("./")
datamodule = VideoDataModule(
    train_root="ucf6/train",
    val_root="ucf6/val",
    batch_size=4,
    num_workers=4,
    num_timesteps=8,
    preprocess_input_size=224,
    preprocess_clip_duration=1,
    preprocess_means=backbone.mean,
    preprocess_stds=backbone.std,
    preprocess_min_short_side=256,
    preprocess_max_short_side=320,
    preprocess_horizontal_flip_p=0.5,
)

head = LinearHead(hidden_size=neck.hidden_size, num_classes=datamodule.num_classes)
model = VideoModel(backbone, head, neck)

Trainer = trainer_factory("single_label_classification")
trainer = Trainer(
    datamodule,
    model,
    max_epochs=8
)

trainer.fit()
  • Perform prediction for a single file or folder of videos:
from video_transformers import VideoModel

model = VideoModel.from_pretrained(model_name_or_path)

model.predict(video_or_folder_path="video.mp4")
>> [{'filename': "video.mp4", 'predictions': {'class1': 0.98, 'class2': 0.02}}]

πŸ€— Full HuggingFace Integration

  • Push your fine-tuned model to the hub:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")

model.push_to_hub('model_name')
  • Load any pretrained video-transformer model from the hub:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")

model.from_pretrained('account_name/model_name')
  • Push your model to HuggingFace hub with auto-generated model-cards:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")
model.push_to_hub('account_name/app_name')
  • (Incoming feature) Push your model as a Gradio app to HuggingFace Space:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")
model.push_to_space('account_name/app_name')

πŸ“ˆ Multiple tracker support

  • Tensorboard tracker is enabled by default.

  • To add Neptune/Layer ... tracking:

from video_transformers.tracking import NeptuneTracker
from accelerate.tracking import WandBTracker

trackers = [
    NeptuneTracker(EXPERIMENT_NAME, api_token=NEPTUNE_API_TOKEN, project=NEPTUNE_PROJECT),
    WandBTracker(project_name=WANDB_PROJECT)
]

trainer = Trainer(
    datamodule,
    model,
    trackers=trackers
)

πŸ•ΈοΈ ONNX support

  • Convert your trained models into ONNX format for deployment:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")
model.to_onnx(quantize=False, opset_version=12, export_dir="runs/exports/", export_filename="model.onnx")

πŸ€— Gradio support

  • Convert your trained models into Gradio App for deployment:
from video_transformers import VideoModel

model = VideoModel.from_pretrained("runs/exp/checkpoint")
model.to_gradio(examples=['video.mp4'], export_dir="runs/exports/", export_filename="app.py")

Contributing

Before opening a PR:

  • Install required development packages:
pip install -e ."[dev]"
  • Reformat with black and isort:
python -m tests.run_code_style format

More Repositories

1

yolov5-pip

Packaged version of ultralytics/yolov5 + many extra features
Python
284
star
2

content-moderation-deep-learning

Deep learning based content moderation from text, audio, video & image input modalities.
268
star
3

craft-text-detector

Packaged, Pytorch-based, easy to use, cross-platform version of the CRAFT text detector
Python
240
star
4

streamlit-image-comparison

Image comparison slider component for Streamlit
Python
193
star
5

small-object-detection-benchmark

icip2022 paper: sahi benchmark on visdrone and xview datasets using fcos, vfnet and tood detectors
Python
138
star
6

pywhisper

openai/whisper + extra features
Python
93
star
7

balanced-loss

Easy to use class balanced cross entropy and focal loss implementation for Pytorch
Python
74
star
8

midv500

Download and convert MIDV-500 annotations to COCO instance segmentation format
Python
73
star
9

ultralyticsplus

Huggingface utilities for Ultralytics/YOLOv8
Python
72
star
10

flask-redis-docker

A minimal template for dockerized flask app with redis task queue
Python
56
star
11

instafake-dataset

Dataset for Intagram Fake and Automated Account Detection
Python
45
star
12

face-recognition-app-tutorial

A face recognition web app powered by Facenet model using Flask, OpenCV, Heroku
HTML
35
star
13

mmdetection-object-tracker

A lightweight script for performing Kalman filter based object tracking using MMDetection models.
Python
22
star
14

augmented-maskrcnn

Object detection and instance segmentation on MaskRCNN with torchvision, albumentations, tensorboard and cocoapi. Supports custom coco datasets with positive/negative samples.
Python
19
star
15

confplot

Confusion Matrix in Python: Plot a pretty confusion matrix (like Matlab) in python using seaborn and matplotlib
Python
9
star
16

face-detection-app-tutorial

A face detection web app powered by SSD face detecctor using Flask, OpenCV, Heroku
Jupyter Notebook
7
star
17

yolov5-to-supervisely

Use your yolov5 predictions as supervisely annotations
Python
4
star
18

ieee-fraud-detection

IEEE Fraud Detection with XGBoost and CatBoost
Jupyter Notebook
4
star
19

turkish-qa-datasets

creating this repo to host some turkish nlp datasets
3
star
20

earth2-scraper

Up-to-date earth2.io data
Python
3
star
21

musicalpy

Easiest way of combining a music and a video
Python
2
star
22

insta-assist

Personal Instagram Tools
Python
2
star
23

cifar100-resnet

ResNet Implementation for CIFAR100 in Pytorch
Jupyter Notebook
2
star
24

fcakyon

2
star
25

deprem-uydu-bina-tespiti

Instance segmentation ve change detection ile uydu goruntusunden bina tespiti
Python
2
star
26

gpt2-shakespeare

A tutorial on GPT2 language model training with texts from Shakespeare
Jupyter Notebook
1
star
27

pytorchvideo

A deep learning library for video understanding research.
Python
1
star
28

glassdoor-review-textgenrnn

Train char-rnn with Glassdoor reviews and generate sentences
Python
1
star
29

DiyarMobileFood

C#
1
star
30

public-files

personal repo for hosting large files
1
star
31

quqixun

1
star