• Stars
    star
    230
  • Rank 174,053 (Top 4 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created almost 5 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Dataset, code and model for the CVPR'20 paper "The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction". And for the ECCV'20 SimAug paper.

Multiverse

This repository contains the code and models for the following CVPR'20 paper:

The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, Alexander Hauptmann

You can find more information at our Project Page and the blog.

The SimAug (ECCV'20) project is here.

  • [11/2022] CMU server is down. You can replace all https://next.cs.cmu.edu with https://precognition.team/next/ to download necessary resources (I did not back up the pretrained models. Please open an issue and share with me the model tgzs if you have downloaded them from the CMU server before).

If you find this code useful in your research then please cite

@inproceedings{liang2020garden,
  title={The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction},
  author={Liang, Junwei and Jiang, Lu and Murphy, Kevin and Yu, Ting and Hauptmann, Alexander},
  booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year={2020}
}
@inproceedings{liang2020simaug,
  title={SimAug: Learning Robust Representations from Simulation for Trajectory Prediction},
  author={Liang, Junwei and Jiang, Lu and Hauptmann, Alexander},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  month = {August},
  year={2020}
}

Introduction

This paper proposes the first multi-future pedestrian trajectory prediction dataset and a multi-future prediction method called Multiverse.

This paper studies the problem of predicting the distribution over multiple possible future paths of people as they move through various visual scenes. We make two main contributions. The first contribution is a new dataset called the Forking Paths Dataset, created in a realistic 3D simulator, which is based on real world trajectory data, and then extrapolated by human annotators to achieve different latent goals. This provides the first benchmark for quantitative evaluation of the models to predict multi-future trajectories.

The Forking Paths Dataset

  • Current dataset version: v1

  • Download links: Google Drive (sha256sum) / Baidu Pan (提取码: tpd7)

  • The dataset includes 3000 1920x1080 videos (750 human-annotated trajectory samples in 4 camera views) with bounding boxes and scene semantic segmentation ground truth. More notes and instructions about the dataset can be found here.

  • Instructions of how to add more human annotations, edit the scenes, recreate from real-world videos, or just simply to play with 3D simulator, can be found here.

  • Instructions of how to semi-automatically re-create real-world videos' scenarios with homography matrices into 3D simulation can be found here.

The Multiverse Model


Our second contribution is a new model to generate multiple plausible future trajectories, which contains novel designs of using multi-scale location encodings and convolutional RNNs over graphs. We refer to our model as Multiverse.

Dependencies

  • Python 2/3; TensorFlow-GPU >= 1.15.0

Pretrained Models

You can download pretrained models by running the script bash scripts/download_single_models.sh.

Testing and Visualization

Instructions for testing pretrained models can be found here.

Qualitative analysis between Social-GAN (left) and our model.

Training new models

Instructions for training new models can be found here.

Acknowledgments

The Forking Paths Dataset is created based on the CARLA Simulator and Unreal Engine 4.

More Repositories

1

awesome_lists

Awesome Lists for Tenure-Track Assistant Professors and PhD students. (助理教授/博士生生存指南)
Python
953
star
2

Object_Detection_Tracking

Out-of-the-box code and models for CMU's object detection and tracking system for multi-camera surveillance videos. Speed optimized Faster-RCNN model. Tensorflow based. Also supports EfficientDet. WACVW'20
Python
407
star
3

social-distancing-prediction

Out-of-the-box code and models for social distancing early forecasting.
Python
40
star
4

FVTA_MemexQA

Real-world photo sequence question answering system (MemexQA). CVPR'18 and TPAMI'19
Python
33
star
5

VERA_Shooter_Localization

Code and model for the Video Event Reconstruction and Analysis (VERA) system. ACM Multimedia.
PHP
26
star
6

DualAttentionNetwork

This repository contains the tensorflow implementation and models for DAN - CVPR 2017 paper
Python
23
star
7

aicity_action

Code and model for the AI City Challenge (CVPR 2022) Track 3 Action Detection (Naturalistic Driving Action Recognition)
Python
18
star
8

VERA_3D_Reconstruction

Code and model for the Video Event Reconstruction and Analysis (VERA) system. This repo is for the 3D event reconstruction part.
JavaScript
12
star
9

MultiTrain

Code and model for "Multi-dataset Training of Transformers for Robust Action Recognition", NeurIPS 2022 Spotlight
11
star
10

junweiliang.github.io

My academic personal page. Written in plain html with bootstrap.css
HTML
11
star
11

ForkingPaths

Dataset from the paper "The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction"
9
star
12

about

My academic personal page. Written in plain html with jQuery.js and bootstrap.css
HTML
7
star
13

Lecture_Attendance_Management

A simple, efficient, mobile-usable web application for lecture attendance management.
PHP
6
star
14

MemexQA_StarterCode

Python
4
star
15

Semantic_Features

Given a list of videos, output the semantic features for each video.
Python
3
star
16

junweilab

Website for Our Lab at HKUST (Guangzhou)
HTML
2
star
17

gunshotDetection

C
2
star
18

videoSync

Python
1
star
19

visual_twitter

Python
1
star