OmniAnomaly
Anomaly Detection for Multivariate Time Series through Modeling Temporal Dependence of Stochastic Variables
OmniAnomaly is a stochastic recurrent neural network model which glues Gated Recurrent Unit (GRU) and Variational auto-encoder (VAE), its core idea is to learn the normal patterns of multivariate time series and uses the reconstruction probability to do anomaly judgment.
Getting Started
Clone the repo
git clone https://github.com/smallcowbaby/OmniAnomaly && cd OmniAnomaly
Get data
SMD (Server Machine Dataset) is in folder ServerMachineDataset
.
You can get the public datasets (SMAP and MSL) using:
wget https://s3-us-west-2.amazonaws.com/telemanom/data.zip && unzip data.zip && rm data.zip
cd data && wget https://raw.githubusercontent.com/khundman/telemanom/master/labeled_anomalies.csv
Install dependencies (with python 3.5, 3.6)
(virtualenv is recommended)
pip install -r requirements.txt
Preprocess the data
python data_preprocess.py <dataset>
where <dataset>
is one of SMAP
, MSL
or SMD
.
Run the code
python main.py
If you want to change the default configuration, you can edit ExpConfig
in main.py
or overwrite the config in main.py
using command line args. For example:
python main.py --dataset='MSL' --max_epoch=20
Data
Dataset Information
Dataset name | Number of entities | Number of dimensions | Training set size | Testing set size | Anomaly ratio(%) |
---|---|---|---|---|---|
SMAP | 55 | 25 | 135183 | 427617 | 13.13 |
MSL | 27 | 55 | 58317 | 73729 | 10.72 |
SMD | 28 | 38 | 708405 | 708420 | 4.16 |
SMAP and MSL
SMAP (Soil Moisture Active Passive satellite) and MSL (Mars Science Laboratory rover) are two public datasets from NASA.
For more details, see: https://github.com/khundman/telemanom
SMD
SMD (Server Machine Dataset) is a new 5-week-long dataset. We collected it from a large Internet company. This dataset contains 3 groups of entities. Each of them is named by machine-<group_index>-<index>
.
SMD is made up by data from 28 different machines, and the 28 subsets should be trained and tested separately. For each of these subsets, we divide it into two parts of equal length for training and testing. We provide labels for whether a point is an anomaly and the dimensions contribute to every anomaly.
Thus SMD is made up by the following parts:
- train: The former half part of the dataset.
- test: The latter half part of the dataset.
- test_label: The label of the test set. It denotes whether a point is an anomaly.
- interpretation_label: The lists of dimensions contribute to each anomaly.
concatenate
Processing
With the default configuration, main.py
follows these steps:
- Train the model with training set, and validate at a fixed frequency. Early stop method is applied by default.
- Test the model on both training set and testing set, and save anomaly score in
train_score.pkl
andtest_score.pkl
. - Find the best F1 score on the testing set, and print the results.
- Init POT model on
train_score
to find the threshold of anomaly score, and using this threshold to predict on the testing set.
Training loss
The figure below are the training loss of our model on MSL and SMAP, which indicates that our model can converge well on these two datasets.