Memory-augmented Dense Predictive Coding for Video Representation Learning
This repository contains the implementation of Memory-augmented Dense Predictive Coding (MemDPC).
Links: [arXiv] [PDF] [Video] [Project page]
News
-
2020/09/08: upload evaluation code for action classification and upload pretrained weights on Kinetics400.
-
2020/08/26: correct the DynamoNet statistics in the figure. DynamoNet uses 500K videos from Youtube8M but only use 10-second clip from each, totally the video length is about 58 days.
Preparation
This repository is implemented in PyTorch 1.2, but newer version should also work. Additionally, it needs cv2, joblib, tqdm, tensorboardX.
For the dataset, please follow the instructions here.
Self-supervised training (MemDPC)
-
Change directory
cd memdpc/ -
Train MemDPC on UCF101 rgb stream
python main.py --gpu 0,1 --net resnet18 --dataset ucf101 --batch_size 16 --img_dim 128 --epochs 500
- Train MemDPC on Kinetics400 rgb stream
python main.py --gpu 0,1,2,3 --net resnet34 --dataset k400 --batch_size 16 --img_dim 224 --epochs 200
Evaluation
Finetune entire network for action classification on UCF101:
-
Change directory
cd eval/ -
Train action classifier by finetuning the pretrained weights
python test.py --gpu 0,1 --net resnet34 --dataset ucf101 --batch_size 16 \
--img_dim 224 --epochs 500 --train_what ft --schedule 300 400
- Train action classifier by freezing the pretrained weights and only a linear layer
python test.py --gpu 0,1 --net resnet34 --dataset ucf101 --batch_size 16 \
--img_dim 224 --epochs 100 --train_what last --schedule 60 80 --dropout 0.5
MemDPC pretrained weights
Citation
If you find the repo useful for your research, please consider citing our paper:
@InProceedings{Han20,
author = "Tengda Han and Weidi Xie and Andrew Zisserman",
title = "Memory-augmented Dense Predictive Coding for Video Representation Learning",
booktitle = "European Conference on Computer Vision",
year = "2020",
}
For any questions, welcome to create an issue or contact Tengda Han ([email protected]).