• Stars
    star
    251
  • Rank 161,862 (Top 4 %)
  • Language
    Jupyter Notebook
  • License
    MIT License
  • Created about 7 years ago
  • Updated almost 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

code for R-C3D

R-C3D: Region Convolutional 3D Network for Temporal Activity Detection

By Huijuan Xu, Abir Das and Kate Saenko (Boston University).

Introduction

We propose a fast end-to-end Region Convolutional 3D Network (R-C3D) for activity detection in continuous video streams. The network encodes the frames with fully-convolutional 3D filters, proposes activity segments, then classifies and refines them based on pooled features within their boundaries.

License

R-C3D is released under the MIT License (refer to the LICENSE file for details).

Citing R-C3D

If you find R-C3D useful in your research, please consider citing:

@inproceedings{Xu2017iccv,
    title = {R-C3D: Region Convolutional 3D Network for Temporal Activity Detection},
    author = {Huijuan Xu and Abir Das and Kate Saenko},
    booktitle = {Proceedings of the International Conference on Computer Vision (ICCV)},
    year = {2017}
}

We build this repo based on Faster R-CNN, C3D and ActivityNet dataset. Please cite the following papers as well:

Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. "Faster R-CNN: Towards real-time object detection with region proposal networks." In Advances in neural information processing systems, pp. 91-99. 2015.

Tran, Du, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. "Learning spatiotemporal features with 3d convolutional networks." In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497. 2015.

Caba Heilbron, Fabian, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. "Activitynet: A large-scale video benchmark for human activity understanding." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970. 2015.

Contents

  1. Installation
  2. Preparation
  3. Training
  4. Testing

Installation:

  1. Clone the R-C3D repository.

    git clone --recursive [email protected]:VisionLearningGroup/R-C3D.git
  2. Build Caffe3d with pycaffe (see: Caffe installation instructions).

    Note: Caffe must be built with Python support!

    cd ./caffe3d
    
    # If have all of the requirements installed and your Makefile.config in place, then simply do:
    make -j8 && make pycaffe
  3. Build R-C3D lib folder.

    cd ./lib    
    make

Preparation:

  1. Download the ground truth annatations and videos in ActivityNet dataset.

    cd ./preprocess/activityNet/
    
    # Download the groud truth annotations in ActivityNet dataset.
    wget http://ec2-52-11-11-89.us-west-2.compute.amazonaws.com/files/activity_net.v1-3.min.json
    
    # Download the videos in ActivityNet dataset into ./preprocess/activityNet/videos.
    python download_video.py
  2. Extract frames from downloaded videos in 25 fps.

    # training video frames are saved in ./preprocess/activityNet/frames/training/
    # validation video frames are saved in ./preprocess/activityNet/frames/validation/ 
    python generate_frames.py
  3. Generate the pickle data for training and testing R-C3D model.

    # generate training data
    python generate_roidb_training.py
    # generate validation data
    python generate_roidb_validation.py

Training:

  1. Download C3D classification pretrain model to ./pretrain/ .

    The C3D model weight pretrained on Sports1M and finetuned on ActivityNet dataset is provided in: caffemodel .

  2. In R-C3D root folder, run:

    ./experiments/activitynet/script_train.sh

Testing:

  1. Download one sample R-C3D model to ./snapshot/ .

    One R-C3D model on ActivityNet dataset is provided in: caffemodel .

    The provided R-C3D model has the Average-mAP 14.4% on the validation set.

  2. In R-C3D root folder, generate the prediction log file on the validation set.

    ./experiments/activitynet/test/script_test.sh
  3. Generate the results.json file from the prediction log file.

    cd ./experiments/activitynet/test
    python activitynet_log_analysis.py test_log_<iters>.txt.*
  4. Get the detection evaluation result.

    cd ./experiments/activitynet/test/Evaluation
    python get_detection_performance.py data/activity_net.v1-3.min.json ../results.json

Notes:

The codes for THUMOS'14 dataset and Charades dataset are uploaded to the corresponding folders.

More Repositories

1

DA_Detection

Implementation of "Strong-Weak Distribution Alignment for Adaptive Object Detection"
Python
345
star
2

SSDA_MME

Semi-supervised Domain Adaptation via Minimax Entropy
Python
295
star
3

CORAL

Correlation Alignment for Domain Adaptation
186
star
4

VisionLearningGroup.github.io

CSS
183
star
5

taskcv-2017-public

Python
168
star
6

DAL

Domain agnostic learning with disentangled representations
Python
135
star
7

DANCE

repository for Universal Domain Adaptation through Self-supervision
Python
123
star
8

caption-guided-saliency

Supplementary material to "Top-down Visual Saliency Guided by Captions" (CVPR 2017)
Python
105
star
9

OVANet

Repository for OVANet
Python
82
star
10

visda-2019-public

Python
58
star
11

OP_Match

Python
56
star
12

visda-2018-public

Python
45
star
13

visda21-dev

Python
45
star
14

Text-to-Clip_Retrieval

Implementation for "Multilevel Language and Vision Integration for Text-to-Clip Retrieval"
Jupyter Notebook
44
star
15

CDS

CDS: Cross-Domain Self-supervised Pre-training
Python
26
star
16

Ask_Attend_and_Answer

Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
C++
23
star
17

JEDDi-Net

Implementation for "Joint Event Detection and Description in Continuous Video Streams"
Jupyter Notebook
22
star
18

UDA_PoseEstimation

Python
19
star
19

MULE

Implementation of "MULE: Multimodal Universal Language Embedding"
Python
14
star
20

Benchmark_Domain_Transfer

Python
13
star
21

SND

Python
13
star
22

Domain2Vec

7
star
23

taskcv_2018_public

Python
4
star
24

mind_back

repository for a paper, Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detection
Python
4
star
25

MMVD

Multimodal Video Description
3
star
26

visda-2018

HTML
3
star
27

SANE

Pytorch implementation of our ECCV paper
Python
2
star
28

visda-2019

CSS
1
star
29

Learning-Similarity-Conditions

Python
1
star
30

Ground2Sky

1
star