• Stars
    star
    453
  • Rank 96,573 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created about 10 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Simple alerting system for Graphite metrics

graphite-beacon

logo

Simple alerting system for Graphite metrics.

Features:

  • Simple installation
  • No software dependencies (Databases, AMQP and etc)
  • Light and fully asynchronous
  • SMTP, HipChat, Slack, PagerDuty, HTTP handlers (PRs for additional handlers are welcome!)
  • Easily configurable and supports historical values

Build status Coverage Version License Downloads

Example:

{
"graphite_url": "http://g.server.org",
"smtp": {
    "from": "[email protected]",
    "to": ["[email protected]"]
},
"alerts": [
    {   "name": "MEM",
        "format": "bytes",
        "query": "aliasByNode(sumSeriesWithWildcards(collectd.*.memory.{memory-free,memory-cached}, 3), 1)",
        "rules": ["critical: < 200MB", "warning: < 400MB", "warning: < historical / 2"] },
    {   "name": "CPU",
        "format": "percent",
        "query": "aliasByNode(sumSeriesWithWildcards(collectd.*.cpu-*.cpu-user, 2), 1)",
        "rules": ["critical: >= 80%", "warning: >= 70%"] }
]}

Requirements

  • python (2.7, 3.3, 3.4)
  • tornado
  • funcparserlib
  • pyyaml

Installation

Python package

graphite-beacon can be installed using pip:

pip install graphite-beacon

Debian package

Using the command line, add the following to your /etc/apt/sources.list system config file:

echo "deb http://dl.bintray.com/klen/deb /" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://dl.bintray.com/klen/deb /" | sudo tee -a /etc/apt/sources.list

Install the package using apt-get:

apt-get update
apt-get install graphite-beacon

Ansible role

There is an ansible role to install the package: https://github.com/Stouts/Stouts.graphite-beacon

Docker

Build a config.json file and run :

docker run -v /path/to/config.json:/srv/alerting/etc/config.json deliverous/graphite-beacon

Usage

Just run graphite-beacon:

$ graphite-beacon
[I 141025 11:16:23 core:141] Read configuration
[I 141025 11:16:23 core:55] Memory (10minute): init
[I 141025 11:16:23 core:166] Loaded with options:
...

Configuration


Time units:

'2second', '3.5minute', '4hour', '5.2day', '6week', '7month', '8year'

short formats are: '2s', '3m', '4.1h' ...

Value units:

short: '2K', '3Mil', '4Bil', '5Tri'

bytes: '2KB', '3MB', '4GB'

bits: '2Kb', '3Mb', '4Gb'

bps: '2Kbps', '3Mbps', '4Gbps'

time: '2s', '3m', '4h', '5d'

The default options are:

Note: comments are not allowed in JSON, but graphite-beacon strips them

    {
        // Graphite server URL
        "graphite_url": "http://localhost",

        // Public graphite server URL
        // Used when notifying handlers, defaults to graphite_url
        "public_graphite_url": null,

        // HTTP AUTH username
        "auth_username": null,

        // HTTP AUTH password
        "auth_password": null,

        // Path to a pidfile
        "pidfile": null,

        // Default values format (none, bytes, s, ms, short)
        // Can be redefined for each alert.
        "format": "short",

        // Default query interval
        // Can be redefined for each alert.
        "interval": "10minute",

        // Default time window for Graphite queries
        // Defaults to query interval, can be redefined for each alert.
        "time_window": "10minute",

        // Notification repeat interval
        // If an alert is failed, its notification will be repeated with the interval below
        "repeat_interval": "2hour",

        // Default end time for Graphite queries
        // Defaults to the current time, can be redefined for each alert.
        "until": "0second",

        // Default loglevel
        "logging": "info",

        // Default method (average, last_value, sum, minimum, maximum).
        // Can be redefined for each alert.
        "method": "average",

        // Default alert to send when no data received (normal = no alert)
        // Can be redefined for each alert
        "no_data": "critical",

        // Default alert to send when loading failed (timeout, server error, etc)
        // (normal = no alert)
        // Can be redefined for each alert
        "loading_error": "critical"

        // Default prefix (used for notifications)
        "prefix": "[BEACON]",

        // Default handlers (log, smtp, hipchat, http, slack, pagerduty)
        "critical_handlers": ["log", "smtp"],
        "warning_handlers": ["log", "smtp"],
        "normal_handlers": ["log", "smtp"],

        // Send initial values (Send current values when reactor starts)
        "send_initial": true,

        // used together to ignore the missing value
        "default_nan_value": -1,
        "ignore_nan": false,

        // Default alerts (see configuration below)
        "alerts": [],

        // Path to other configuration files to include
        "include": []
    }

You can setup options with a configuration file. See examples for JSON and YAML.

A config.json file in the same directory that you run graphite-beacon from will be used automatically.

Setup alerts

Currently two types of alerts are supported:

  • Graphite alert (default) - check graphite metrics
  • URL alert - load http and check status

Note: comments are not allowed in JSON, but graphite-beacon strips them

  "alerts": [
    {
      // (required) Alert name
      "name": "Memory",

      // (required) Alert query
      "query": "*.memory.memory-free",

      // (optional) Alert type (graphite, url)
      "source": "graphite",

      // (optional) Default values format (none, bytes, s, ms, short)
      "format": "bytes",

      // (optional) Alert method (average, last_value, sum, minimum, maximum)
      "method": "average",

      // (optional) Alert interval [eg. 15second, 30minute, 2hour, 1day, 3month, 1year]
      "interval": "1minute",

      // (optional) What kind of alert to send when no data received (normal = no alert)
      "no_data": "warning",

      // (optional) Alert interval end time (see "Alert interval" for examples)
      "until": "5second",

      // (required) Alert rules
      // Rule format: "{level}: {operator} {value}"
      // Level one of [critical, warning, normal]
      // Operator one of [>, <, >=, <=, ==, !=]
      // Value (absolute value: 3000000 or short form like 3MB/12minute)
      // Multiple conditions can be separated by AND or OR conditions
      "rules": [ "critical: < 200MB", "warning: < 300MB" ]
    }
  ]
Historical values

graphite-beacon supports "historical" values for a rule. For example you may want to get warning when CPU usage is greater than 150% of normal usage:

"warning: > historical * 1.5"

Or memory is less than half the usual value:

"warning: < historical / 2"

Historical values for each query are kept. A historical value represents the average of all values in history. Rules using a historical value will only work after enough values have been collected (see history_size).

History values are kept for 1 day by default. You can change this with the history_size option.

See the below example for how to send a warning when today's new user count is less than 80% of the last 10 day average:

alerts: [
  {
    "name": "Registrations",
    // Run once per day
    "interval": "1day",
    "query": "Your graphite query here",
    // Get average for last 10 days
    "history_size": "10day",
    "rules": [
      // Warning if today's new user less than 80% of average for 10 days
      "warning: < historical * 0.8",
     // Critical if today's new user less than 50% of average for 10 days
      "critical: < historical * 0.5"
    ]
  }
],

Handlers

Handlers allow for notifying an external service or process of an alert firing.

Email Handler

Sends an email (enabled by default).

{
    // SMTP default options
    "smtp": {
        "from": "beacon@graphite",
        "to": [],                   // List of email addresses to send to
        "host": "localhost",        // SMTP host
        "port": 25,                 // SMTP port
        "username": null,           // SMTP user (optional)
        "password": null,           // SMTP password (optional)
        "use_tls": false,           // Use TLS?
        "html": true,               // Send HTML emails?

        // Graphite link for emails (By default is equal to main graphite_url)
        "graphite_url": null
    }
}

HipChat Handler

Sends a message to a HipChat room.

{
    "hipchat": {
        // (optional) Custom HipChat URL
        "url": 'https://api.custom.hipchat.my',

        "room": "myroom",
        "key": "mykey"
    }
}

Webhook Handler (HTTP)

Triggers a webhook.

{
    "http": {
        "url": "http://myhook.com",
        "params": {},                 // (optional) Additional query(data) params
        "method": "GET"               // (optional) HTTP method
    }
}

Slack Handler

Sends a message to a user or channel on Slack.

{
    "slack": {
        "webhook": "https://hooks.slack.com/services/...",
        "channel": "#general",          // #channel or @user (optional)
        "username": "graphite-beacon",
    }
}

Command Line Handler

Runs a command.

{
    "cli": {
        // Command to run (required)
        // Several variables that will be substituted by values are allowed:
        //  ${level} -- alert level
        //  ${name} -- alert name
        //  ${value} -- current metrics value
        //  ${limit_value} -- metrics limit value
        "command": "./myscript ${level} ${name} ${value} ...",

        // Whitelist of alerts that will trigger this handler (optional)
        // All alerts will trigger this handler if absent.
        "alerts_whitelist": ["..."]
    }
}

PagerDuty Handler

Triggers a PagerDuty incident.

{
    "pagerduty": {
        "subdomain": "yoursubdomain",
        "apitoken": "apitoken",
        "service_key": "servicekey",
    }
}

Telegram Handler

Sends a Telegram message.

{
    "telegram": {
        "token": "telegram bot token",
        "bot_ident": "token you choose to activate bot in a group"
        "chatfile": "path to file where chat ids are saved, optional field"
    }
}

Command Line Usage

  $ graphite-beacon --help
  Usage: graphite-beacon [OPTIONS]

  Options:

    --config                         Path to an configuration file (JSON/YAML)
                                     (default config.json)
    --graphite_url                   Graphite URL (default http://localhost)
    --help                           show this help information
    --pidfile                        Set pid file

    --log_file_max_size              max size of log files before rollover
                                     (default 100000000)
    --log_file_num_backups           number of log files to keep (default 10)
    --log_file_prefix=PATH           Path prefix for log files. Note that if you
                                     are running multiple tornado processes,
                                     log_file_prefix must be different for each
                                     of them (e.g. include the port number)
    --log_to_stderr                  Send log output to stderr (colorized if
                                     possible). By default use stderr if
                                     --log_file_prefix is not set and no other
                                     logging is configured.
    --logging=debug|info|warning|error|none
                                     Set the Python log level. If 'none', tornado
                                     won't touch the logging configuration.
                                     (default info)

Bug tracker

If you have any suggestions, bug reports or annoyances please report them to the issue tracker at https://github.com/klen/graphite-beacon/issues

Contributors

License

Licensed under a MIT license

If you wish to express your appreciation for the role, you are welcome to send a postcard to:

Kirill Klenov
pos. Severny 8-3
MO, Istra, 143500
Russia

More Repositories

1

pylama

Code audit tool for python.
Python
1,048
star
2

mixer

Mixer -- Is a fixtures replacement. Supported Django, Flask, SqlAlchemy and custom python objects.
Python
942
star
3

py-frameworks-bench

Another benchmark for some python frameworks
Python
712
star
4

muffin

Muffin is a fast, simple and asyncronous web-framework for Python 3
Python
671
star
5

django_markdown

Django markdown support and wysiwig
JavaScript
391
star
6

peewee_migrate

Simple migration engine for Peewee
Python
353
star
7

nvim-test

A Neovim wrapper for running tests
Lua
181
star
8

Flask-Foundation

Quick start with Flask
Python
154
star
9

nvim-config-local

Secure load local config files for neovim
Lua
150
star
10

aioauth-client

OAuth client for aiohttp
Python
141
star
11

atmark

Awk+Sed for humans
Python
69
star
12

python-scss

Python scss parser.
Python
67
star
13

dealer

Make some staff
Python
62
star
14

marshmallow-peewee

Peewee ORM integration with the marshmallow (de)serialization library.
Python
59
star
15

zeta-library

Css, scss, js parser and linker. Also framework for working with static files
Python
52
star
16

Flask-Collect

Collect static files in flask application
Python
49
star
17

makesite

makesite is a collection of scripts for deploying and managing web projects
Python
47
star
18

flask-pw

Peewee ORM integration for Flask framework
Python
45
star
19

peewee-aio

Async support for Peewee ORM
Python
45
star
20

adrest

Another django rest framework
Python
42
star
21

.vim

my vim settings
Vim Script
39
star
22

django-netauth

django auth backend
Python
36
star
23

asgi-tools

Tools to build ASGI apps
Python
35
star
24

pomodoro-tracker-locales

Language files
24
star
25

asgi-babel

Adds internationalization (i18n) support to ASGI applications (Asyncio/Trio)
Python
24
star
26

http-router

A simple router for HTTP applications
Python
24
star
27

rope-vim

Pathogen compatable ropevim plugin. Dont need install rope libs in system.
Python
21
star
28

bottle-peewee

Integrate Peewee ORM to Bottle framework
Python
19
star
29

muffin-admin

Admin interface for Muffin Framework
Python
18
star
30

mahjong.horneds.com

Riichi Mahjong Scores Trainer
CoffeeScript
18
star
31

flask-restler

Yet another REST library for Flask
Python
15
star
32

unite-radio.vim

Play radio stations in your VIM
Vim Script
15
star
33

aio-databases

Async Support for various databases
Python
14
star
34

django-gitrevision

Django git revision, simple add current git revision to request object for use in tempaltes and views.
Python
14
star
35

imgproxy

Python support for ImgProxy image processing server (https://imgproxy.net)
Python
14
star
36

pytest-aio

Is a simple pytest plugin for testing async python code
Python
12
star
37

klen.github.io

My github powered site.
HTML
12
star
38

muffin-rest

REST helpers for Muffin Framework
Python
12
star
39

aio-peewee

Tools to make Peewee work when using Asyncio
Python
11
star
40

muffin-jinja2

Jinja2 templates for Muffin framework
Python
11
star
41

donald

Make asyncio great again
Python
10
star
42

pypika-orm

Async ORM based on PyPika
Python
9
star
43

bottle-login

Implement users' sessions in Bottle framework
Python
9
star
44

asgi-prometheus

Support Prometheus metrics for ASGI applications
Python
9
star
45

muffin-peewee

Peewee integration to Muffin framework
Python
9
star
46

asgi-sessions

Signed Cookie-Based HTTP sessions for ASGI applications
Python
8
star
47

muffin-session

Session for Muffin Framework
Python
7
star
48

muffin-mongo

MongoDB support for Muffin framework
Python
7
star
49

starter

Create the skeleton for new projects.
Python
6
star
50

muffin-redis

Redis support for Muffin framework
Python
6
star
51

muffin-sentry

Sentry integration to Muffin Framework.
Python
5
star
52

pyserve

Serve local dirs (human version)
Python
5
star
53

muffin-example

Example Muffin application
Python
5
star
54

dotfiles

kk .dotfiles / use it for your own risks
Shell
4
star
55

bottle-jade

Provide Jade templates for Bottle framework
Python
4
star
56

django-gishelper

Useful commands for geodjango
Python
4
star
57

muffin-oauth

OAuth1/2 support for Muffin framework.
Python
4
star
58

muffin-databases

Async support for a range of databases for Muffin Framework
Python
4
star
59

knocker

A self contained service to make HTTP calls
Python
4
star
60

muffin-debugtoolbar

Debug Toolbar for Muffin applications
JavaScript
4
star
61

muffin-babel

Extension to Muffin that adds localization support with help of babel.
Python
3
star
62

inirama

Simple INI parser
Python
3
star
63

pylama_pylint

Pylint support for pylama.
Makefile
3
star
64

starlette-plugins

Create Starlette Plugins easier
Python
3
star
65

example_tornadio_project

Sources for http://klen.github.com/tornadio_socket-io-ru.html
Python
3
star
66

muffin-peewee-aio

Peewee integration to Muffin framework with async support
Python
2
star
67

redux-axios-reducers

Redux Reducers for Axios
CoffeeScript
2
star
68

pytest-redislite

Pytest plugin for testing code using Redi
Python
2
star
69

bottle-manage

Script manager for bottle framework.
Python
2
star
70

muffin-grpc

GRPC Support for Muffin Framework
Python
2
star
71

fquest

ZeroQuest lazy RPG. Moscow Facebook Hackday.
Python
2
star
72

aio-apiclient

Simple Asyncio Client for any HTTP APIs
Python
2
star
73

muffin-jade

Jade templates for Muffin Framework
Python
2
star
74

signalbus

Simple and small library to broadcast signals with typing support
Python
2
star
75

muffin-metrics

Send application metrics to Graphite
Python
2
star
76

hvim

Haskell mode for vim
Vim Script
2
star
77

Flask-jsonrpc-example

Some flask experements
Python
2
star
78

starlette-views

A helper to make views faster with Starlette
Python
2
star
79

hydrogenjs

Simple MVC system for atomjs
JavaScript
1
star
80

simpletree

Fastest and simplest tree implementations for Django
Python
1
star
81

filler

Simple game on javascript and canvas.
JavaScript
1
star
82

modconfig

Simple hierarchic configuration manager for apps
Python
1
star
83

sailplay

Python client for API sailplay.ru
Python
1
star
84

zeta-libs

Frameworks repo for zetalibrary
JavaScript
1
star
85

redux-code

Yet another creators library
TypeScript
1
star
86

tweetchi

Python
1
star
87

muffin-prometheus

Prometheus metrics exporter for Muffin framework
Python
1
star
88

fl

Forbidden lands russian generator
TypeScript
1
star