• Stars
    star
    729
  • Rank 62,157 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 6 years ago
  • Updated 8 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

KDD 2019: Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network

OmniAnomaly

Anomaly Detection for Multivariate Time Series through Modeling Temporal Dependence of Stochastic Variables

OmniAnomaly is a stochastic recurrent neural network model which glues Gated Recurrent Unit (GRU) and Variational auto-encoder (VAE), its core idea is to learn the normal patterns of multivariate time series and uses the reconstruction probability to do anomaly judgment.

Getting Started

Clone the repo

git clone https://github.com/smallcowbaby/OmniAnomaly && cd OmniAnomaly

Get data

SMD (Server Machine Dataset) is in folder ServerMachineDataset.

You can get the public datasets (SMAP and MSL) using:

wget https://s3-us-west-2.amazonaws.com/telemanom/data.zip && unzip data.zip && rm data.zip

cd data && wget https://raw.githubusercontent.com/khundman/telemanom/master/labeled_anomalies.csv

Install dependencies (with python 3.5, 3.6)

(virtualenv is recommended)

pip install -r requirements.txt

Preprocess the data

python data_preprocess.py <dataset>

where <dataset> is one of SMAP, MSL or SMD.

Run the code

python main.py

If you want to change the default configuration, you can edit ExpConfig in main.py or overwrite the config in main.py using command line args. For example:

python main.py --dataset='MSL' --max_epoch=20

Data

Dataset Information

Dataset name Number of entities Number of dimensions Training set size Testing set size Anomaly ratio(%)
SMAP 55 25 135183 427617 13.13
MSL 27 55 58317 73729 10.72
SMD 28 38 708405 708420 4.16

SMAP and MSL

SMAP (Soil Moisture Active Passive satellite) and MSL (Mars Science Laboratory rover) are two public datasets from NASA.

For more details, see: https://github.com/khundman/telemanom

SMD

SMD (Server Machine Dataset) is a new 5-week-long dataset. We collected it from a large Internet company. This dataset contains 3 groups of entities. Each of them is named by machine-<group_index>-<index>.

SMD is made up by data from 28 different machines, and the 28 subsets should be trained and tested separately. For each of these subsets, we divide it into two parts of equal length for training and testing. We provide labels for whether a point is an anomaly and the dimensions contribute to every anomaly.

Thus SMD is made up by the following parts:

  • train: The former half part of the dataset.
  • test: The latter half part of the dataset.
  • test_label: The label of the test set. It denotes whether a point is an anomaly.
  • interpretation_label: The lists of dimensions contribute to each anomaly.

concatenate

Processing

With the default configuration, main.py follows these steps:

  • Train the model with training set, and validate at a fixed frequency. Early stop method is applied by default.
  • Test the model on both training set and testing set, and save anomaly score in train_score.pkl and test_score.pkl.
  • Find the best F1 score on the testing set, and print the results.
  • Init POT model on train_score to find the threshold of anomaly score, and using this threshold to predict on the testing set.

Training loss

The figure below are the training loss of our model on MSL and SMAP, which indicates that our model can converge well on these two datasets.

image image

More Repositories

1

donut

WWW 2018: Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications
Python
463
star
2

TraceAnomaly

ISSRE'20: Unsupervised Detection of Microservice Trace Anomalies through Service-Level Deep Bayesian Networks
Python
313
star
3

LogParse

An adaptive log template extraction toolkit.
Python
218
star
4

LogClass

IEEE-TNSM 2021: Anomalous Log Identification and Classification with Partial Labels
Python
167
star
5

Log2Vec

A distributed representation method for online logs.
Roff
161
star
6

Squeeze

ISSRE 2019: Generic and Robust Localization of Multi-Dimensional Root Cause
Python
96
star
7

KPI-Anomaly-Detection

2018AIOps: The 1st match for AIOps
80
star
8

TraceRCA

Practical Root Cause Localization for Microservice Systems via Trace Analysis. IWQoS 2021
Python
74
star
9

CIRCA

Causal Inference-based Root Cause Analysis
Python
73
star
10

DejaVu

Code and datasets for FSE'22 paper "Actionable and Interpretable Fault Localization for Recurring Failures in Online Service Systems"
Jupyter Notebook
73
star
11

AIOps-Challenge-2020-Data

The published dataset of AIOps Challenge 2020
62
star
12

Bagel

IPCCC 2018: Robust and Unsupervised KPI Anomaly Detection Based on Conditional Variational Autoencoder
Python
50
star
13

JumpStarter

Python
44
star
14

PSqueeze

Python
30
star
15

MultiDimension-Localization

2019AIOps: The 2nd match for AIOps
23
star
16

TraceVAE

The source code for "Unsupervised Anomaly Detection on Microservice Traces through Graph VAE" in WWW2023.
Python
18
star
17

OpsEval-Datasets

Datasets for OpsEval
Python
15
star
18

CTF_data

Data of paper "CTF: Anomaly Detection in High-Dimensional Time Series with Coarse-to-Fine Model Transfer"
13
star
19

DOMI_code

code for DOMI
Python
11
star
20

kontrast

Python
9
star
21

aiops2020-judge

AIOps2020评测脚本
Python
7
star
22

CMDiagnostor

Python
7
star
23

DOMI_dataset

DOMI dataset
7
star
24

AutoKAD

Python
6
star
25

RC-LIR

Python
5
star
26

GTrace

Source code for GTrace (ESEC/FSE'23 industry track).
Python
4
star
27

KAD-Disformer

Python
3
star
28

AnoTuner

Python
3
star
29

PreFix

SIGMETRICS 2018: PreFix: Switch Failure Prediction in Datacenter Networks
2
star
30

course.aiops.org

HTML
2
star
31

aiops-2022-judge

2022挑战赛评测脚本
Python
2
star
32

Chain-of-Event

Python
2
star
33

DejaVu-Omni

Code and datasets for TOSEM paper "DejaVu-Omni: Actionable, Robust and Interpretable Fault Localization for Recurring Failures in Online Service Systems"
Jupyter Notebook
1
star
34

AlertRCA

Python
1
star
35

OpenCompass-OpsQA

Python
1
star