Note RF100 is part of the CVinW challenge!
Roboflow 100 πΈ : A Rich, Multi-Domain Object Detection Benchmark
This repository implements the Roboflow 100 benchmark developed by Roboflow. It contains code to download the dataset and reproduce mAP values for YOLOv5 and YOLOv7 Fine-Tuning and GLIP Evaluation on 100 of Roboflow Universe datasets.
RF100 was sponsored with
RF100 paper is on arxiv, if you used RF100 in your research, please consider citing us
@misc{2211.13523,
Author = {Floriana Ciaglia and Francesco Saverio Zuppichini and Paul Guerrie and Mark McQuade and Jacob Solawetz},
Title = {Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
Year = {2022},
Eprint = {arXiv:2211.13523},
}
RF100
RF100 contains the following datasets, carefully chosen from more than 90'000 datasets hosted on our universe hub. The datasets are splitted in 7 categories: Aerial, Videogames, Microscopic, Underwater, Documents, Electromagnetic and Real World.
| Category | Datasets | Images | Classes |
|---|---|---|---|
| Aerial | 7 | 9683 | 24 |
| Videogames | 7 | 11579 | 88 |
| Microscopic | 11 | 13378 | 28 |
| Underwater | 5 | 18003 | 39 |
| Documents | 8 | 24813 | 90 |
| Electromagnetic | 12 | 36381 | 41 |
| Real World | 50 | 110615 | 495 |
| Total | 100 | 224,714 | 805 |
We provide a notebook to help you using RF100 with PyTorch
Getting Started
First, clone this repo and go inside it.
git clone https://github.com/roboflow-ai/roboflow-100-benchmark.git
cd roboflow-100-benchmark
git submodule update --init --recursiveYou will need an API key. RF100 can be accessed with any key from Roboflow, head over our doc to learn how to get one.
Then, export the key to your current shell
export ROBOFLOW_API_KEY=<YOUR_API_KEY>Note: The datasets are taken from datasets_links.txt, you can modify that file to add/remove datasets.
Docker
The easiest and faster way to download RF100 is using docker and our Dockerfile.
NOTE Be sure to do the post process steps after you installed docker, we will read and write to shared volumes so your user should also be in the docker group.
If you have an NVIDIA GPU, be sure to also install nvidia docker
Be sure to have the ROBOFLOW_API_KEY in your env, then run it (we are passing our user so you shouldn't have permission errors on rf100 folder)
docker run --rm -it \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v $(pwd)/rf100:/workspace/rf100 \
-v /etc/group:/etc/group:ro \
-u "$(id -u):$(id -g)" \
roboflow/rf100-downloadroboflow/rf100-download is hosted on our docker hub
Internally, RF100 will downloaded to /app/rf100. You can also specify the format with the -f flag, by default coco is used.
docker run --rm -it \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/rf100:/workspace/rf100 \
-v /etc/group:/etc/group:ro \
-u "$(id -u):$(id -g)" \
roboflow/rf100-download -f yolov5If you want to build the container and not use the image on the hub, run
docker build -t rf100-download -f Dockerfile.rf100.download .Local Env
To download RF100 in your local enviroment (python >=3.6), you need to install roboflow
pip install roboflowThen,
chmod 770 ./scripts/download_datasets.sh
./scripts/download_datasets.sh
./scripts/download_datasets.sh -f yolov5 $ change format
./scripts/download_datasets.sh -l <path_to_my_location> change download locationFormats
Supported formats are
cocoyolov5(used by YOLOv7 as well)
Reproduce Results
We will use docker to ensure the same enviroment is used.
First, build the container
docker build -t rf100-benchmark -f Dockerfile.rf100.benchmark .
Then, follow the guide for each model.
All results are stored inside ./runs.
Warning The run must be uninterrupted otherwise the downloaded datasets may be unfinished or corrupted
Warning If you have
permission deniederrors, you can either removerunswithsudo rm -rf ./runsor give permissions to everybodysudo chmod -R +x ./runs.
YOLOv5 Fine-Tuning
Note, we will map the current folder to the container file system to persist data
mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./yolov5-benchmark/train.shYOLOv7 Fine-Tuning
Note, we will map the current folder to the container file system to persist data
mkdir -p runs &&
nvidia-docker run --gpus all --rm -d --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./yolov7-benchmark/train.shGLIP
mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./GLIP-benchmark/train.shNote We create a notebook to analysis why and where GLIP did bad and good
COCO evaluation
If you are interesting in only the test annotations for rf100, e.g. you are using pycocotools to compute maps, we have create a zip file with root/<dataset_name>/<test_annotations>.json. We hope this will make it easier for you to evaluate your models.
Appendix
Credits
We thank all the authors of the original datasets, below a table linking the Roboflow 100 dataset's name to its original counterpart