[Pre-print] [Website]

Achal Dave, Pavel Tokmakov, Deva Ramanan

1. Download models and extract them to release/models
2. Install pytorch 0.4.0.
3. Run git submodule update --init.
4. Setup detectron-pytorch.
5. Setup flownet2. If you just want to use the appearance stream, you can skip this step.
6. Install requirements with pip install -r requirements.txt¹.
7. Copy ./release/example_config.yaml to ./release/config.yaml, and edit fields marked with ***EDIT THIS***.
8. Add root directory to PYTHONPATH: source ./env.sh activate.

All scripts needed for running our models on standard datasets, as well as on new videos, are provided in the ./release directory. Outside of the release directory, this repository contains a number of scripts which are not used for the final results. They can be safely ignored, but are provided in case anyone finds them useful.

1. Extract frames: To run the model on your own video, first dump the frames from your video. For a single video, you can just use

   ffmpeg -i video.mp4 %04d.jpg

   Alternatively, you can use this script to extract frames in parallel on multiple videos.

2. Run joint model: To run the joint model, run the following commands:

   # Inputs
   FRAMES_DIR=/path/to/frames/dir
   # Outputs
   OUTPUT_DIR=/path/to/output/dir
   
   python release/custom/run.py \
   --model joint \
   --frames-dir ${FRAMES_DIR} \
   --output-dir ${OUTPUT_DIR}

3. Run appearance only model: To run only the appearance model, you don't need to compute optical flow, or set up flownet2:

   python release/custom/run.py \
   --model appearance \
   --frames-dir ${FRAMES_DIR} \
   --output-dir ${OUTPUT_DIR}

The instructions for FBMS, DAVIS 2016/2017 and YTVOS datasets are roughly the same. Once you have downloaded the dataset and edited the paths in ./release/config.yaml, run the following scripts:

# or davis16, davis17, ytvos
dataset=fbms
python release/${dataset}/compute_flow.py
python release/${dataset}/infer.py
python release/${dataset}/track.py
# For evaluation:
python release/${dataset}/evaluate.py

Note that by default, we use our final model trained on COCO, FlyingThings3D, DAVIS, and YTVOS. For YTVOS, we provide the option to run using a model that was trained without YTVOS, to evaluate generalization. To activate this, pass --without-ytvos-train to release/ytvos/infer.py and release/ytvos/track.py.

1: This should contain all the requirements, but this was created manually so I may be missing some pip modules. If you run into an import error, try pip installing the module, and/or file an issue.