TSB-UAD

An End-to-End Benchmark Suite for Univariate Time-Series Anomaly Detection

TSB-UAD is a new open, end-to-end benchmark suite to ease the evaluation of univariate time-series anomaly detection methods. Overall, TSB-UAD contains 12686 time series with labeled anomalies spanning different domains with high variability of anomaly types, ratios, and sizes. Specifically, TSB-UAD includes 18 previously proposed datasets containing 1980 time series from real-world data science applications. Motivated by flaws in certain datasets and evaluation strategies in the literature, we study anomaly types and data transformations to contribute two collections of datasets. Specifically, we generate 958 time series using a principled methodology for transforming 126 time-series classification datasets into time series with labeled anomalies. In addition, we present a set of data transformations with which we introduce new anomalies in the public datasets, resulting in 10828 time series (92 datasets) with varying difficulty for anomaly detection.

If you use TSB-UAD in your project or research, cite the following two papers:

• VLDB 2022a
• VLDB 2022b

References

"TSB-UAD: An End-to-End Benchmark Suite for Univariate Time-Series Anomaly Detection"
John Paparrizos, Yuhao Kang, Paul Boniol, Ruey Tsay, Themis Palpanas, and Michael Franklin.
Proceedings of the VLDB Endowment (PVLDB 2022) Journal, Volume 15, pages 1697–1711

@article{paparrizos2022tsb,
  title={Tsb-uad: an end-to-end benchmark suite for univariate time-series anomaly detection},
  author={Paparrizos, John and Kang, Yuhao and Boniol, Paul and Tsay, Ruey S and Palpanas, Themis and Franklin, Michael J},
  journal={Proceedings of the VLDB Endowment},
  volume={15},
  number={8},
  pages={1697--1711},
  year={2022},
  publisher={VLDB Endowment}
}

"Volume Under the Surface: A New Accuracy Evaluation Measure for Time-Series Anomaly Detection"
John Paparrizos, Paul Boniol, Themis Palpanas, Ruey Tsay, Aaron Elmore, and Michael Franklin
Proceedings of the VLDB Endowment (PVLDB 2022) Journal, Volume 15, pages 2774‑2787

@article{paparrizos2022volume,
  title={{Volume Under the Surface: A New Accuracy Evaluation Measure for Time-Series Anomaly Detection}},
  author={Paparrizos, John and Boniol, Paul and Palpanas, Themis and Tsay, Ruey S and Elmore, Aaron and Franklin, Michael J},
  journal={Proceedings of the VLDB Endowment},
  volume={15},
  number={11},
  pages={2774--2787},
  year={2022},
  publisher={VLDB Endowment}
}

"Local Evaluation of Time Series Anomaly Detection Algorithms", Accepted in KDD 2022 Research Track: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Affiliation Metrics

Contributors

• John Paparrizos (University of Chicago)
• Yuhao Kang (University of Chicago)
• Alex Wu (University of Chicago)
• Teja Bogireddy (University of Chicago)
• Paul Boniol (Université Paris Cité)

Installation

The following tools are required to install TSB-UAD from source:

• git
• conda (anaconda or miniconda)

Steps

1. First, due to limitations in the upload size on GitHub, we host the datasets at a different location. Please download the datasets using the following links:

• Public: https://www.thedatum.org/datasets/TSB-UAD-Public.zip

• Synthetic: https://www.thedatum.org/datasets/TSB-UAD-Synthetic.zip

• Artificial: https://www.thedatum.org/datasets/TSB-UAD-Artificial.zip

• • The UCR classification datasets used to generate the Artificial datasets: https://www.thedatum.org/datasets/UCR2022_DATASETS.zip

2. Clone this repository using git and change into its root directory.

git clone https://github.com/johnpaparrizos/TSB-UAD.git
cd TSB-UAD/

3. Create and activate a conda-environment 'TSB'.

conda env create --file environment.yml
conda activate TSB

4. Install the dependencies from requirements.txt:

pip install -r requirements.txt

5. Install TSB-UAD:

pip install .

Benchmark

All are datasets and time series are stored in ./data. We describe below the different types of datasets used in our benchmark.

Benchmark: Public Real Datasets

We first collected 18 public datasets from heterogeneous domains (you may find them in benchmark/). The figure below shows some typical outliers in these 18 datasets.

Briefly, TSB-UAD includes the following datasets:

The following table summarizes different characteristics of the datasets.

You may find more details (and the references) in our paper.

Benchmark: Synthetic and Artificial Datasets

We also include synthetic and artifical datasets. We construct and transformed dataset based on the UCR archive (you may find them in UCR2018-NEW/ which contains 128 subfolders). The synthetic and artifical datasets are stored here:

• artificial/ contains the data that are constructed based on UCR2018-NEW. Below is a figure illustrating the construction process.

• synthetic/ contains the data that are synthesized by local and global tranformations. Below is a figure illustrating the transformation process.

Anomaly Detectors

We use 13 anomaly detection methods proposed for univariate time series. the following table lists and describes the methods considered in our benchmark:

You may find more details (and the references) in our paper. In italics are methods that are available but not evaluated yet.

Overall Results

We report in the following table the average AUC-ROC and F score for each method on each dataset. The accuracy scores are averaged for all the time series in each dataset.

Here we only report the average accuracy (AUC-ROC and F score) for each dataset. You may find here the accuracy tables for each time series and each accuracy measure.

We then compare the accuracy scores of the anoamly detection methods on time series containing point versus sequence anomalies and a single versus multiple anomalies. The following figure summarize the results:

This figure and many others comparisons are described (and reproductible) in this notebook, summarized in this document. The scripts to reproduce the critical diagrams and the statistical tests are here.

Usage

Anomaly Detector

We depicts below a code snippet demonstrating how to use one anomaly detector (in this example, IForest).

import os
import pandas as pd
from TSB_UAD.core import tsb_uad


df = pd.read_csv('data/benchmark/ECG/MBA_ECG805_data.out', header=None).to_numpy()
data = df[:, 0].astype(float)
label = df[:, 1]
results = tsb_uad(data, label, 'IForest', metric='all')

metrics = {}
for metric in results.keys():
    metrics[metric] = [results[metric]]
    print(metric, ':', results[metric])

df = pd.DataFrame(data=metrics)
df.to_csv(os.path.join('./all_metrics.csv'))

AUC_ROC : 0.9216216369841076
AUC_PR : 0.6608577550833885
Precision : 0.7342093339374717
Recall : 0.4010891089108911
F : 0.5187770129662238
Precision_at_k : 0.4010891089108911
Rprecision : 0.7486112853253205
Rrecall : 0.3097733542316151
RF : 0.438214653167952
R_AUC_ROC : 0.989123018780308
R_AUC_PR : 0.9435238401582703
VUS_ROC : 0.9734357459251715
VUS_PR : 0.8858037295594041
Affiliation_Precision : 0.9630674176380548
Affiliation_Recall : 0.9809813654809071

You may find more details on how to run each anomaly detection method in this notebook and this script.

Build Artifical Dataset

You may find details and examples on how to create artifical datasets in this notebook and this script

Apply Transformations to a Dataset

You may find details and examples on how to apply transformations on a dataset in this notebook and this script