loglizer

Loglizer is a machine learning-based log analysis toolkit for automated anomaly detection.

Loglizer是一款基于AI的日志大数据分析工具, 能用于自动异常检测、智能故障诊断等场景

Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information during system operation that allows developers and support engineers to monitor their systems and track abnormal behaviors and errors. Loglizer provides a toolkit that implements a number of machine-learning based log analysis techniques for automated anomaly detection.

🔭 If you use loglizer in your research for publication, please kindly cite the following paper.

• Shilin He, Jieming Zhu, Pinjia He, Michael R. Lyu. Experience Report: System Log Analysis for Anomaly Detection, IEEE International Symposium on Software Reliability Engineering (ISSRE), 2016. [Bibtex][中文版本] (ISSRE Most Influential Paper)

Framework

The log analysis framework for anomaly detection usually comprises the following components:

1. Log collection: Logs are generated at runtime and aggregated into a centralized place with a data streaming pipeline, such as Flume and Kafka.
2. Log parsing: The goal of log parsing is to convert unstructured log messages into a map of structured events, based on which sophisticated machine learning models can be applied. The details of log parsing can be found at our logparser project.
3. Feature extraction: Structured logs can be sliced into short log sequences through interval window, sliding window, or session window. Then, feature extraction is performed to vectorize each log sequence, for example, using an event counting vector.
4. Anomaly detection: Anomaly detection models are trained to check whether a given feature vector is an anomaly or not.

Models

Anomaly detection models currently available:

Log data

We have collected a set of labeled log datasets in loghub for research purposes. If you are interested in the datasets, please follow the link to submit your access request.

Install

git clone https://github.com/logpai/loglizer.git
cd loglizer
pip install -r requirements.txt

API usage

# Load HDFS dataset. If you would like to try your own log, you need to rewrite the load function.
(x_train, y_train), (x_test, y_test) = dataloader.load_HDFS(...)

# Feature extraction and transformation
feature_extractor = preprocessing.FeatureExtractor()
feature_extractor.fit_transform(...) 

# Model training
model = PCA()
model.fit(...)

# Feature transform after fitting
x_test = feature_extractor.transform(...)
# Model evaluation with labeled data
model.evaluate(...)

# Anomaly prediction
x_test = feature_extractor.transform(...)
model.predict(...) # predict anomalies on given data

For more details, please follow the demo in the docs to get started. Please note that all ML models are not magic, you need to figure out how to tune the parameters in order to make them work on your own data.

Benchmarking results

If you would like to reproduce the following results, please run benchmarks/HDFS_bechmark.py on the full HDFS dataset (HDFS100k is for demo only).

Contributors

• Shilin He, The Chinese University of Hong Kong
• Jieming Zhu, The Chinese University of Hong Kong, currently at Huawei Noah's Ark Lab
• Pinjia He, The Chinese University of Hong Kong, currently at ETH Zurich

Feedback

For any questions or feedback, please post to the issue page.

History

• May 14, 2016: initial commit
• Sep 21, 2017: update code and readme
• Mar 21, 2018: rewrite most of the code and add detailed comments
• Feb 18, 2019: restructure the repository with hands-on demo