Perception Test: A Diagnostic Benchmark for Multimodal Video Models

News

Test Splits are now available for all the challenges! For Downloads see download section.

Join the first Perception Test challenge organised as an ICCV2023 workshop, website here ptchallenge-workshop.github.io.

Overview

Perception Test: A Diagnostic Benchmark for Multimodal Video Models is a multimodal benchmark designed to comprehensively evaluate the perception and reasoning skills of multimodal video models. The Perception Test dataset introduces real-world videos designed to show perceptually interesting situations and defines multiple tasks (object and point tracking, action and sound localisation, multiple-choice and grounded video question-answering) that require understanding of memory, abstract patterns, physics, and semantics, across visual, audio, and text modalities.

In this repository, you will find:

• A summary of the Perception Test and the associated challenge
• A detailed description of the data and annotations in the Perception Test (interactive demo notebook here)
• Details about how to download the data and annotations in the Perception Test (download section here)
• Metrics for evaluating the performance on the different tasks (metrics section here)
• Dummy baselines showcasing how to evaluate models on each of the tasks (baselines section here)

5-minutes summary of the Perception Test

Try the Perception Test for yourself by accessing this quiz.

For more example videos in the Perception Test, check out this playlist.

Download the data and annotations

Test annotations will be released August 1st as part of the challenge phases.

The Perception Test dataset can be downloaded as zip files containing:

• annotations in JSON format
• videos (including audio) as MP4 files
• audio-only files in WAV format
• pre-computed features for the action localisation and sound localisation tasks.

Links

Challenge Downloads

Multi-Choice vQA
Challenge link: https://eval.ai/web/challenges/challenge-page/2091/overview

Single Object Tracking
Challenge link: https://eval.ai/web/challenges/challenge-page/2094/overview

Temporal Action Localisation
Challenge link: https://eval.ai/web/challenges/challenge-page/2101/overview

Baselines

In this repo we provide dummy baselines to demonstrate how to load the data, evaluate and recreate some baseline results from the paper. For the other results in the baselines section in the paper, we will be adding another external repo.

Metrics

Metrics code to evaluate performance for the different tasks coming soon.

Perception Test annotations

Explore the annotations: data_visualisation.ipynb

Summary

Video metadata

Object tracks

Point tracks

Action segments

Sound segments

Multiple-choice video question-answers

Grounded video question-answers

Feedback and support

If you have any questions, feedback, or require support regarding the Perception Test dataset or challenge, please contact us at [email protected].

Citing this work

@misc{patraucean2023perception,
      title={Perception Test: A Diagnostic Benchmark for Multimodal Video Models}, 
      author={Viorica Pătrăucean and Lucas Smaira and Ankush Gupta and Adrià Recasens Continente and Larisa Markeeva and Dylan Banarse and Skanda Koppula and Joseph Heyward and Mateusz Malinowski and Yi Yang and Carl Doersch and Tatiana Matejovicova and Yury Sulsky and Antoine Miech and Alex Frechette and Hanna Klimczak and Raphael Koster and Junlin Zhang and Stephanie Winkler and Yusuf Aytar and Simon Osindero and Dima Damen and Andrew Zisserman and João Carreira},
      year={2023},
      eprint={2305.13786},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License and disclaimer

Copyright 2022 DeepMind Technologies Limited

All software is licensed under the Apache License, Version 2.0 (Apache 2.0); you may not use this file except in compliance with the Apache 2.0 license. You may obtain a copy of the Apache 2.0 license at: https://www.apache.org/licenses/LICENSE-2.0

All other materials are licensed under the Creative Commons Attribution 4.0 International License (CC-BY). You may obtain a copy of the CC-BY license at: https://creativecommons.org/licenses/by/4.0/legalcode

Unless required by applicable law or agreed to in writing, all software and materials distributed here under the Apache 2.0 or CC-BY licenses are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the licenses for the specific language governing permissions and limitations under those licenses.

This is not an official Google product.