RabbitMQ Prometheus.io exporter

Implemented as RabbitMQ Management Plugin plugin. Also exports Erlang VM and process metrics (~ 100 metrics in total).

Implemented using Erlang Prometheus.io client

Versioning

While RabbitMQ transitions from webmachine to cowboy we maintain two branches one for 3.6.x and one for 3.7.x. Plugin version should be read as follows: 3.7.1.x - where 3.7.1 is required RabbitMQ version and x is just incremental version of the plugin.

Installation

Download suitable version and follow regular RabbitMQ plugin installation instructions.

 rabbitmq-plugins enable prometheus_rabbitmq_exporter

If you are running on Linux/FreeBSD/Mac, you may find prometheus_process_exporter useful:

rabbitmq-plugins enable prometheus_process_collector

Prometheus process collector.

Troubleshooting

`undef` error

If you see something like this:

{could_not_start,rabbitmq_management,
       {undef,
           [{prometheus_http,setup,[],[]}

I.e. undef error mentioning a module starting with prometheus_, chances you forgot to enable a plugin (see #27 for example).

Module `prometheus_process_collector` is unloadable

{plugin_module_unloadable,"prometheus_process_collector",
                             {error,on_load_failure}}

{error,{load_failed,"Failed to load NIF library:
 '/<...>/plugins/prometheus_process_collector-1.1.0/priv/prometheus_process_collector.so:
 failed to map segment from shared object'"}}

Prometheus process collector uses NIFs underneath and failed to load shared object in module on_load callback. Please check that RABBITMQ_PLUGINS_EXPAND_DIR doesn't have noexec flag set (see #26).

Crashes with something like {error,{bad_lib,"Library version (2.11) not compatible (with 2.10)."}}

This means erl_nif version prometheus_process_collector built with differs from yours (see Version Management section). You can rebuild the plugin yourself very easily - clone https://github.com/deadtrickster/prometheus_process_collector.git and run rebar3 archive

Glibc-related errors when `prometheus_process_collector` enabled

prometheus_process_collector plugin comes with prebuilt shared object. And it looks like my Glibc version differs from yours. You can rebuild the plugin yourself very easily - clone https://github.com/deadtrickster/prometheus_process_collector.git and run rebar3 archive

Latest Docker:

docker run -p 8080:15672 deadtrickster/rabbitmq_prometheus

Alpine-based image is also available: docker run -p 8080:15672 deadtrickster/rabbitmq_prometheus:latest-alpine

Configuration

This exporter supports the following options via rabbitmq_exporter entry of prometheus app env:

path - scrape endpoint. Default is "metrics". Note RabbitMQ translates this to "{management_plugin_path_prefix}/api/metrics";
use_mgmt_auth - use built-in management auth. Default is false. If true, relies on management plugin for authentication (that guest:guest on fresh setups);
format - scrape format. Default is prometheus_text_format;
exchange_messages_stat - same as queue_messages_state but for the exchanges;
queue_messages_stat - messages state to export. Default is hopefully reasonable. You can read more about possible values here;
connections_total_enabled - Default is false. If true, the exporter will iterate over all connections and export count grouped by connection state (running, flow, etc).

Sample /etc/rabbitmq/rabbitmq.config showing how to customize the scrape path, and connections_total_enabled:

[
 {rabbit, [
   {loopback_users, []},
   {tcp_listeners, [5672]},
   {ssl_listeners, []}
 ]},
 {prometheus, [
   {rabbitmq_exporter, [
     {path, "/mymetrics"},
     {connections_total_enabled, true}
   ]}
 ]},
 {rabbitmq_management, [
   {listener, [
     {port, 15672},
     {ssl, false}
   ]}
 ]}
].

For the latest list of supported options look here.

Metrics

RabbitMQ Specific Metrics

Overview

rabbitmq_connections
Type: gauge.
RabbitMQ Connections count.
rabbitmq_connections_total (disabled by default)
Type: gauge.
Labels: state.
RabbitMQ connections count grouped by connection state.
rabbitmq_channels
Type: gauge.
RabbitMQ Channels count.
rabbitmq_queues
Type: gauge.
RabbitMQ Queues count.
rabbitmq_exchanges
Type: gauge.
RabbitMQ Exchanges count.
rabbitmq_consumers
Type: gauge.
RabbitMQ Consumers count.
rabbitmq_queues_disk_reads
Type: counter.
Total number of times messages have been read from disk by all queues.
rabbitmq_queues_disk_writes
Type: counter.
Total number of times messages have been written to disk by all queues.
rabbitmq_messages_ready
Type: gauge.
Messages ready for delivery.
rabbitmq_messages_unacknowledged
Type: gauge.
Delivered but unacknowledged messages.
rabbitmq_messages_published_total
Type: counter.
Count of messages published.
rabbitmq_messages_confirmed_total
Type: counter.
Count of messages confirmed.
rabbitmq_messages_delivered_total
Type: counter.
Count of messages delivered in acknowledgement mode to consumers.
rabbitmq_messages_delivered_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode to consumers.
rabbitmq_messages_get_total
Type: counter.
Count of messages delivered in acknowledgement mode in response to basic.get.
rabbitmq_messages_get_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode in response to basic.get.
rabbitmq_messages_deliver_get_total
Type: counter.
Sum of messages_delivered_total, messages_delivered_no_ack_total, messages_get_total and messages_get_no_ack_total.
rabbitmq_messages_redelivered_total
Type: counter.
Count of subset of delivered messages which had the redelivered flag set.
rabbitmq_messages_returned_total
Type: counter.
Count of messages returned to publisher as unroutable.

Queues

Labels: vhost, queue.

rabbitmq_queue_durable
Type: boolean.
Whether or not the queue survives server restarts.
rabbitmq_queue_auto_delete
Type: boolean.
Whether the queue will be deleted automatically when no longer used.
rabbitmq_queue_exclusive
Type: boolean.
True if queue is exclusive (i.e. has owner_pid), false otherwise.
rabbitmq_queue_messages_ready
Type: gauge.
Number of messages ready to be delivered to clients.
rabbitmq_queue_messages_unacknowledged
Type: gauge.
Number of messages delivered to client but not yet acknowledged.
rabbitmq_queue_messages
Type: gauge.
Sum of ready and unacknowledged messages (queue depth).
rabbitmq_queue_messages_ready_ram
Type: gauge.
Number of messages from messages_ready which are resident in ram.
rabbitmq_queue_messages_unacknowledged_ram
Type: gauge.
Number of messages from messages_unacknowledged which are resident in ram.
rabbitmq_queue_messages_ram
Type: gauge.
Total number of messages which are resident in ram.
rabbitmq_queue_messages_persistent
Type: gauge.
Total number of persisted messages in the queue (will always be 0 for transient queues).
rabbitmq_queue_message_bytes
Type: gauge.
Sum of the size of all message bodies in the queue. This does not include the message properties (including headers) or any overhead.
rabbitmq_queue_message_bytes_ready
Type: gauge.
Like message_bytes but counting only those messages ready to be delivered to clients.
rabbitmq_queue_message_bytes_unacknowledged
Type: gauge.
Like message_bytes but counting only those messages delivered to clients but not yet acknowledged.
rabbitmq_queue_message_bytes_ram
Type: gauge.
Like message_bytes but counting only those messages which are in RAM.
rabbitmq_queue_message_bytes_persistent
Type: gauge.
Like message_bytes but counting only those messages which are persistent.
rabbitmq_queue_head_message_timestamp
Type: gauge.
The timestamp property of the first message in the queue, if present. Timestamps of messages only appear when they are in the paged-in state.
rabbitmq_queue_disk_reads
Type: counter.
Total number of times messages have been read from disk by this queue since it started.
rabbitmq_queue_disk_writes
Type: counter.
Total number of times messages have been written to disk by this queue since it started.
rabbitmq_queue_disk_size_bytes
Type: gauge.
Disk space occupied by the queue.
rabbitmq_queue_consumers
Type: gauge.
Number of consumers.
rabbitmq_queue_consumer_utilisation
Type: gauge.
Fraction of the time (between 0.0 and 1.0) that the queue is able to immediately deliver messages to consumers. This can be less than 1.0 if consumers are limited by network congestion or prefetch count.
rabbitmq_queue_memory
Type: gauge.
Bytes of memory consumed by the Erlang process associated with the queue, including stack, heap and internal structures.
rabbitmq_queue_state
Type: gauge.
The state of the queue. NaN if queue is located on cluster nodes that are currently down. 0 if queue is running normally. MsgCount if queue is synchronising.
rabbitmq_queue_messages_published_total
Type: counter.
Count of messages published.
rabbitmq_queue_messages_confirmed_total
Type: counter.
Count of messages confirmed.
rabbitmq_queue_messages_delivered_total
Type: counter.
Count of messages delivered in acknowledgement mode to consumers.
rabbitmq_queue_messages_delivered_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode to consumers.
rabbitmq_queue_messages_get_total
Type: counter.
Count of messages delivered in acknowledgement mode in response to basic.get.
rabbitmq_queue_messages_get_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode in response to basic.get.
rabbitmq_queue_messages_deliver_get_total
Type: counter.
Sum of messages_delivered_total, messages_delivered_no_ack_total, messages_get_total and messages_get_no_ack_total.
rabbitmq_queue_messages_redelivered_total
Type: counter.
Count of subset of delivered messages which had the redelivered flag set.
rabbitmq_queue_messages_returned_total
Type: counter.
Count of messages returned to publisher as unroutable.

Exchanges

Labels: vhost, exchange.

rabbitmq_exchange_messages_published_total
Type: counter.
Count of messages published.
rabbitmq_exchange_messages_published_in_total
Type: counter.
Count of messages published "in" to an exchange, i.e. not taking account of routing.
rabbitmq_exchange_messages_published_out_total
Type: counter.
Count of messages published "out" of an exchange, i.e. taking account of routing.
rabbitmq_exchange_messages_confirmed_total
Type: counter.
Count of messages confirmed.
rabbitmq_exchange_messages_delivered_total
Type: counter.
Count of messages delivered in acknowledgement mode to consumers.
rabbitmq_exchange_messages_delivered_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode to consumers.
rabbitmq_exchange_messages_get_total
Type: counter.
Count of messages delivered in acknowledgement mode in response to basic.get.
rabbitmq_exchange_messages_get_no_ack_total
Type: counter.
Count of messages delivered in no-acknowledgement mode in response to basic.get.
rabbitmq_exchange_messages_deliver_get_total
Type: counter.
Sum of *messages_delivered_total, *messages_delivered_no_ack_total, *messages_get_total and *messages_get_no_ack_total.
rabbitmq_exchange_messages_redelivered_total
Type: counter.
Count of subset of delivered messages which had the redelivered flag set.
rabbitmq_exchange_messages_returned_total
Type: counter.
Count of messages returned to publisher as unroutable.

Mnesia Tables

Various metrics for RabbitMQ-specific Mnesia tables.

Labels: table.

rabbitmq_mnesia_table_read_only
Type: boolean.
Access mode of the table, 1 if table is read_only or 0 otherwise.
rabbitmq_mnesia_table_disc_copies
Type: gauge.
Number of the nodes where a disc_copy of the table resides according to the schema.
rabbitmq_mnesia_table_disc_only_copies
Type: gauge.
Number of the nodes where a disc_only_copy of the table resides according to the schema.
rabbitmq_mnesia_table_local_content
Type: boolean.
If the table is configured to have locally unique content on each node, value is 1 or 0 otherwise.
rabbitmq_mnesia_table_majority_required
Type: boolean.
If 1, a majority of the table replicas must be available for an update to succeed.
rabbitmq_mnesia_table_master_nodes
Type: gauge.
Number of the master nodes of a table.
rabbitmq_mnesia_table_memory_bytes
Type: gauge.
The number of bytes allocated to the table on this node.
rabbitmq_mnesia_table_ram_copies
Type: gauge.
Number of the nodes where a ram_copy of the table resides according to the schema.
rabbitmq_mnesia_table_records_count
Type: gauge.
Number of records inserted in the table.
rabbitmq_mnesia_table_disk_size_bytes
Type: gauge.
Disk space occupied by the table (DCL + DCD).

Nodes

Cluster/nodes metrics.

rabbitmq_node_up
Type: boolean.
Labels: name, type.
Node running status.

Erlang VM & OTP Metrics

System Info

erlang_vm_ets_limit
Type: gauge.
The maximum number of ETS tables allowed.
erlang_vm_logical_processors
Type: gauge.
The detected number of logical processors configured in the system.
erlang_vm_logical_processors_available
Type: gauge.
The detected number of logical processors available to the Erlang runtime system.
erlang_vm_logical_processors_online
Type: gauge.
The detected number of logical processors online on the system.
erlang_vm_port_count
Type: gauge.
The number of ports currently existing at the local node.
erlang_vm_port_limit
Type: gauge.
The maximum number of simultaneously existing ports at the local node.
erlang_vm_process_count
Type: gauge.
The number of processes currently existing at the local node.
erlang_vm_process_limit
Type: gauge.
The maximum number of simultaneously existing processes at the local node.
erlang_vm_schedulers
Type: gauge.
The number of scheduler threads used by the emulator.
erlang_vm_schedulers_online
Type: gauge.
The number of schedulers online.
erlang_vm_smp_support
Type: boolean.
1 if the emulator has been compiled with SMP support, otherwise 0.
erlang_vm_threads
Type: boolean.
1 if the emulator has been compiled with thread support, otherwise 0.
erlang_vm_thread_pool_size
Type: gauge.
The number of async threads in the async thread pool used for asynchronous driver calls.
erlang_vm_time_correction
Type: boolean.
1 if time correction is enabled, otherwise 0.

Statistics

erlang_vm_statistics_bytes_output_total
Type: counter.
The total number of bytes output to ports.
erlang_vm_statistics_bytes_received_total
Type: counter.
The total number of bytes received through ports.
erlang_vm_statistics_context_switches
Type: counter.
The total number of context switches since the system started.
erlang_vm_statistics_garbage_collection_number_of_gcs
Type: counter.
The total number of garbage collections since the system started.
erlang_vm_statistics_garbage_collection_bytes_reclaimed
Type: counter.
The total number of bytes reclaimed by GC since the system started.
erlang_vm_statistics_garbage_collection_words_reclaimed
Type: counter.
The total number of words reclaimed by GC since the system started.
erlang_vm_statistics_reductions_total
Type: counter.
Total reductions count.
erlang_vm_statistics_run_queues_length_total
Type: gauge.
The total length of the run-queues. That is, the number of processes and ports that are ready to run on all available run-queues.
erlang_vm_statistics_runtime_milliseconds
Type: counter.
The sum of the runtime for all threads in the Erlang runtime system.
erlang_vm_statistics_wallclock_time_milliseconds
Type: counter.
Can be used in the same manner as erlang_vm_statistics_runtime_milliseconds, except that real time is measured as opposed to runtime or CPU time.

Memory

erlang_vm_memory_atom_bytes_total{usage="free|used"}
Type: gauge.
The total amount of memory currently allocated for atoms. This memory is part of the memory presented as system memory.
erlang_vm_memory_bytes_total{kind="system|processes"}
Type: gauge.
The total amount of memory currently allocated. This is the same as the sum of the memory size for processes and system.
erlang_vm_dets_tables
Type: gauge.
Erlang VM DETS Tables count.
erlang_vm_ets_tables
Type: gauge.
Erlang VM ETS Tables count.
erlang_vm_memory_processes_bytes_total{usage="free|used"}
Type: gauge.
The total amount of memory currently allocated for the Erlang processes.
erlang_vm_memory_system_bytes_total{usage="atom|binary|code|ets|other"}
Type: gauge.
The total amount of memory currently allocated for the emulator that is not directly related to any Erlang process. Memory presented as processes is not included in this memory.

Mnesia

erlang_mnesia_held_locks
Type: gauge.
Number of held locks.
erlang_mnesia_lock_queue
Type: gauge.
Number of transactions waiting for a lock.
erlang_mnesia_transaction_participants
Type: gauge.
Number of participant transactions.
erlang_mnesia_transaction_coordinators
Type: gauge.
Number of coordinator transactions.
erlang_mnesia_failed_transactions
Type: counter.
Number of failed (i.e. aborted) transactions.
erlang_mnesia_committed_transactions
Type: gauge.
Number of committed transactions.
erlang_mnesia_logged_transactions
Type: counter.
Number of transactions logged.
erlang_mnesia_restarted_transactions
Type: counter.
Total number of transaction restarts.

Process Metrics

(Process info collector must be enabled)

process_open_fds
Type: gauge.
Number of open file descriptors.
process_max_fds
Type: gauge.
Maximum number of open file descriptors.
process_start_time_seconds
Type: gauge.
Start time of the process since unix epoch in seconds.
process_uptime_seconds
Type: gauge.
Process uptime in seconds.
process_threads_total
Type: gauge.
Process threads count.
process_virtual_memory_bytes
Type: gauge.
Virtual memory size in bytes.
process_resident_memory_bytes
Type: gauge.
Resident memory size in bytes;
process_cpu_seconds_total{kind="utime|stime"}
Type: counter.
Process CPU time.

Exporter Metrics

Labels: registry, content_type.

telemetry_scrape_duration_seconds
Type: summary.
Scrape duration.
telemetry_scrape_size_bytes
Type: summary.
Scrape size, uncompressed.

License

MIT

deadtrickster/prometheus_rabbitmq_exporter

deadtrickster

Reviews

Repository Details

RabbitMQ Prometheus.io exporter

TOC

Versioning

Installation

Troubleshooting

`undef` error

Module `prometheus_process_collector` is unloadable

Crashes with something like {error,{bad_lib,"Library version (2.11) not compatible (with 2.10)."}}

Glibc-related errors when `prometheus_process_collector` enabled

Latest Docker:

Configuration

Metrics

RabbitMQ Specific Metrics

Overview

Queues

Exchanges

Mnesia Tables

Nodes

Erlang VM & OTP Metrics

System Info

Statistics

Memory

Mnesia

Process Metrics

Exporter Metrics

License

More Repositories

deadtrickster/prometheus_rabbitmq_exporter

deadtrickster

Reviews

Repository Details

RabbitMQ Prometheus.io exporter

TOC

Versioning

Installation

Troubleshooting

undef error

Module prometheus_process_collector is unloadable

Crashes with something like {error,{bad_lib,"Library version (2.11) not compatible (with 2.10)."}}

Glibc-related errors when prometheus_process_collector enabled

Latest Docker:

Configuration

Metrics

RabbitMQ Specific Metrics

Overview

Queues

Exchanges

Mnesia Tables

Nodes

Erlang VM & OTP Metrics

System Info

Statistics

Memory

Mnesia

Process Metrics

Exporter Metrics

License

More Repositories

`undef` error

Module `prometheus_process_collector` is unloadable

Glibc-related errors when `prometheus_process_collector` enabled