Airflow Prometheus Exporter

The Airflow Prometheus Exporter exposes various metrics about the Scheduler, DAGs and Tasks which helps improve the observability of an Airflow cluster.

The exporter is based on this prometheus exporter for Airflow.

Requirements

The plugin has been tested with:

Airflow >= 1.10.4
Python 3.6+

The scheduler metrics assume that there is a DAG named canary_dag. In our setup, the canary_dag is a DAG which has a tasks which perform very simple actions such as establishing database connections. This DAG is used to test the uptime of the Airflow scheduler itself.

Installation

The exporter can be installed as an Airflow Plugin using:

pip install airflow-prometheus-exporter

This should ideally be installed in your Airflow virtualenv.

Metrics

Metrics will be available at

http://<your_airflow_host_and_port>/admin/metrics/

Task Specific Metrics

`airflow_task_status`

Number of tasks with a specific status.

All the possible states are listed here.

`airflow_task_duration`

Duration of successful tasks in seconds.

`airflow_task_fail_count`

Number of times a particular task has failed.

`airflow_xcom_param`

value of configurable parameter in xcom table

xcom fields is deserialized as a dictionary and if key is found for a paticular task-id, the value is reported as a guage

Add task / key combinations in config.yaml:

xcom_params:
  -
    task_id: abc
    key: count
  -
    task_id: def
    key: errors

a task_id of 'all' will match against all airflow tasks:

xcom_params:
 -
    task_id: all
    key: count

Dag Specific Metrics

`airflow_dag_status`

Number of DAGs with a specific status.

All the possible states are listed here

`airflow_dag_run_duration`

Duration of successful DagRun in seconds.

Scheduler Metrics

`airflow_dag_scheduler_delay`

Scheduling delay for a DAG Run in seconds. This metric assumes there is a canary_dag.

The scheduling delay is measured as the delay between when a DAG is marked as SCHEDULED and when it actually starts RUNNING.

`airflow_task_scheduler_delay`

Scheduling delay for a Task in seconds. This metric assumes there is a canary_dag.

`airflow_num_queued_tasks`

Number of tasks in the QUEUED state at any given instance.

robinhood/airflow-prometheus-exporter

robinhood

Reviews

Repository Details