+---------------+------------------+--------------+--------------+
| Queue Name    | Worker Available | Jobs Waiting | Jobs Running |
+---------------+------------------+--------------+--------------+
| check_results | 1                | 0            | 0            |
| eventhandler  | 50               | 0            | 0            |
| host          | 50               | 0            | 1            |
| service       | 50               | 0            | 13           |
+---------------+------------------+--------------+--------------+

+-----------------------+------------------+--------------+--------------+
| Queue Name            | Worker Available | Jobs Waiting | Jobs Running |
+-----------------------+------------------+--------------+--------------+
| check_results         | 1                | 0            | 0            |
| eventhandler          | 50               | 0            | 0            |
| host                  | 50               | 0            | 1            |
| hostgroup_japan       | 3                | 1            | 3            |
| service               | 50               | 0            | 13           |
| servicegroup_selenium | 2                | 0            | 2            |
+-----------------------+------------------+--------------+--------------+

The easiest variant is a simple load balancing. For example if your single Naemon box just cannot handle the load, you could just add a worker in the same network (or even on the same host) to reduce your load on the Naemon box. Therefor we just enable hosts, services and eventhandler on the server and the worker.

Pro:

• reduced load on your monitoring box

Contra:

• no failover

If your checks have to be run from different network segments, then you can use the hostgroups (or servicegroups) to define a hostgroup for specific worker. The general hosts and services queue is disabled for this worker and just the hosts and services from the given hostgroup will be processed.

Pro:

• reduced load on your monitoring box

• ability to access remote networks

Contra:

• no failover

Your distributed setup could easily be extended to a load balanced setup with just adding more worker of the same config.

Pro:

• reduced load on your monitoring box

• ability to access remote networks

• automatic failover and load balancing for worker

Contra:

• no failover for the master

If you just want to replace a current NSCA solution, you could load the Mod-Gearman NEB module and disable all distribution features. You still can receive passive results by the core send via send_gearman / send_multi. Make sure you use the same encryption settings like the neb module or your core won’t be able to process the results or use the accept_clear_results option.

Pro:

• easy to setup in existing environments

In case your network is unstable or you need a gui view from the remote location or any other reason which makes a remote core unavoidable you may want this setup. Thist setup consists of 2 independent Mod-Gearman setups and the slave worker just send their results to the master via the dup_server option. The master objects configuration must contain all slave services and hosts. The configuration sync is not part of Mod-Gearman.

Pro:

• independent from network outtakes

• local view

Contra:

• more complex setup

• requires configuration sync

Sometimes you may need to reverse the direction of the initial connection attempt. Usually the worker and the neb module open the initial connection so they need to access the gearmand port. In cases where no direct connection is possible use ssh tunnel or the Gearman proxy. The Gearman proxy just puts jobs from one gearmand into another gearmand and vice versa.

gearman_proxy.pl can be found at https://github.com/sni/gearman-proxy

Pro:

• changes direction of initial connection setup

• buffers network outages

Contra:

• two more daemon to monitor and maintain

Using OMD is propably the easiest way of installing and using Mod-Gearman. You just have to run omd config or set Mod-Gearman to on.

OMD is available for Debian, Ubuntu, Centos/Redhat and SLES.

It is strongly recommended to use the official packages or the unoffical packages which contains Debian Squeeze and various Ubuntu packages.

The easy and proper way is to build RPM packages. The following steps assume a Centos 5.7. Other releases may have different versions but should behave similar.

Build/install Gearmand rpms

Build/install Mod-Gearman rpms

Finally start and check your installation

Pre Requirements:

• gcc / g++

• autoconf / automake / autoheader

• libtool

• libgearman (>= 0.14)

Download the tarball and perform the following steps:

Then add the mod_gearman_naemon.o to your Naemon installation and add a broker line to your naemon.cfg:

see Configuration for details on all parameters

The next step is to start one or more worker. You may use the same configuration file as for the neb module.

or use the supplied init script.

or a supplied init script. Command line arguments have change in recent gearman versions and you now should use something like:

A sample broker in your naemon.cfg could look like:

See the following list for a detailed explanation of available options:

Shared options for worker and the NEB module:

config

include config from this file. Options are the same as described here. include is an alias for config.

config=/etc/naemon/mod_gm_worker.conf

debug

use debug to increase the verbosity of the module. Possible values are:

• 0 - only errors

• 1-4 - debug verbosity

• 5 - trace and all gearman related logs are going to stdout

Default is 0.

debug=1

logmode

set way of logging. Possible values are:

• automatic - logfile when a logfile is specified. stdout when no logfile is given. stdout for tools.

• stdout - just print all messages to stdout

• syslog - use syslog for all log events

• file - use logfile

• core - use naemon internal loging (not thread safe! Use with care)

Default is automatic.

logmode=automatic

logfile

Path to the logfile.

logfile=/path/to/log.file

server

sets the address of your gearman job server. Can be specified more than once to add more server. Mod-Gearman uses the first server available.

server=localhost:4730,remote_host:4730

eventhandler

defines if the module should distribute execution of eventhandlers.

eventhandler=yes

services

defines if the module should distribute execution of service checks.

services=yes

hosts

defines if the module should distribute execution of host checks.

hosts=yes

hostgroups

sets a list of hostgroups which will go into seperate queues.

hostgroups=name1,name2,name3

servicegroups

sets a list of servicegroups which will go into seperate queues.

servicegroups=name1,name2,name3

encryption

enables or disables encryption. It is strongly advised to not disable encryption. Anybody will be able to inject packages to your worker. Encryption is enabled by default and you have to explicitly disable it. When using encryption, you will either have to specify a shared password with key=... or a keyfile with keyfile=.... Default is On.

encryption=yes

key

A shared password which will be used for encryption of data pakets. Should be at least 8 bytes long. Maximum length is 32 characters.

key=secret

keyfile

The shared password will be read from this file. Use either key or keyfile. Only the first 32 characters from the first line will be used. Whitespace to the right will be trimmed.

keyfile=/path/to/secret.file

use_uniq_jobs

Using uniq keys prevents the gearman queues from filling up when there is no worker. However, gearmand seems to have problems with the uniq key and sometimes jobs get stuck in the queue. Set this option to off when you run into problems with stuck jobs but make sure your worker are running. Default is On.

use_uniq_jobs=on

gearman_connection_timeout

Timeout in milliseconds when connecting to gearmand daemon. Set to 0 to disable the timeout. Default is 5000.

gearman_connection_timeout=5000

Additional options for the NEB module only:

localhostgroups

sets a list of hostgroups which will not be executed by gearman. They are just passed through.

localhostgroups=name1,name2,name3

localservicegroups

sets a list of servicegroups which will not be executed by gearman. They are just passed through.

localservicegroups=name1,name2,name3

queue_custom_variable

Can be used to define the target queue by a custom variable in addition to host/servicegroups. When set for ex. to WORKER you then could define a _WORKER custom variable for your hosts and services to directly set the worker queue. The host queue is inherited unless overwritten by a service custom variable. Set the value of your custom variable to local to bypass Mod-Gearman (Same behaviour as in localhostgroups/localservicegroups).

queue_custom_variable=WORKER

do_hostchecks

Set this to no if you want Mod-Gearman to only take care of servicechecks. No hostchecks will be processed by Mod-Gearman. Use this option to disable hostchecks and still have the possibility to use hostgroups for easy configuration of your services. If set to yes, you still have to define which hostchecks should be processed by either using hosts or the hostgroups option. Default: yes

do_hostchecks=yes

result_workers

Enable or disable result worker thread. The default is one, but you can set it to zero to disabled result workers, for example if you only want to export performance data.

result_workers=0

perfdata

Defines if the module should distribute perfdata to gearman. Can be specified multiple times and accepts comma separated lists.

perfdata=yes

Note	processing of perfdata is not part of mod_gearman. You will need additional worker for handling performance data. For example: PNP4Nagios. Performance data is just written to the gearman queue.

perfdata_send_all

Set perfdata_send_all=yes to submit all performance data of all hosts and services regardless of if they have process_performance_data enabled or not. Default: no

perfdata_send_all=yes

perfdata_mode

There will be only a single job for each host or service when putting performance data onto the perfdata_queue in overwrite mode. In append mode perfdata will be stored as long as there is memory left. Setting this to overwrite helps preventing the perf_data queue from getting to big. Monitor your perfdata carefully when using the append mode. Possible values are:

• 1 - overwrite

• 2 - append

Default is 1.

perfdata_mode=1

result_queue

sets the result queue. Necessary when putting jobs from several Naemon instances onto the same gearman queues. Default: check_results

result_queue=check_results_naemon1

orphan_host_checks

The Mod-Gearman NEB module will submit a fake result for orphaned host checks with a message saying there is no worker running for this queue. Use this option to get better reporting results, otherwise your hosts will keep their last state as long as there is no worker running. Default is yes.

orphan_host_checks=yes

orphan_service_checks

Same like orphan_host_checks but for services. Default is yes.

orphan_service_checks=yes

orphan_return

Set return code of orphaned checks. Possible values are:

• 0 - OK

• 1 - WARNING

• 2 - CRITICAL

• 3 - UNKNOWN

Default is 2.

orphan_return=2

accept_clear_results

When enabled, the NEB module will accept unencrypted results too. This is quite useful if you have lots of passive checks and make use of send_gearman/send_multi where you would have to spread the shared key to all clients using these tools. Default is no.

accept_clear_results=yes

latency_flatten_window

When enabled, reschedules host/service checks if their latency is more than one second. This value is the maximum delay in seconds applied to hosts/services. Set to 0 or less than 0 to disable rescheduling. Default is 30.

latency_flatten_window=30

Additional options for worker:

identifier

Identifier for this worker. Will be used for the worker_identifier queue for status requests. You may want to change it if you are using more than one worker on a single host. Defaults to the current hostname.

identifier=hostname_test

pidfile

Path to the pidfile.

pidfile=/path/to/pid.file

job_timeout

Default job timeout in seconds. Currently this value is only used for eventhandler. The worker will use the values from the core for host and service checks. Default: 60

job_timeout=60

max-age

Threshold for discarding too old jobs. When a new job is older than this amount of seconds it will not be executed and just discarded. This will result in a message like "(Could Not Start Check In Time)". Possible reasons for this are time differences between core and worker (use NTP!) or the smart rescheduler of the core which should be disabled. Set to zero to disable this check. Default: 0

max-age=600

min-worker

Minimum number of worker processes which should run at any time. Default: 1

min-worker=1

max-worker

Maximum number of worker processes which should run at any time. You may set this equal to min-worker setting to disable dynamic starting of workers. When setting this to 1, all services from this worker will be executed one after another. Default: 20

max-worker=20

spawn-rate

Defines the rate of spawned worker per second as long as there are jobs waiting. Default: 1

spawn-rate=1

load_limit1

Set a limit based on the 1min load average. When exceding the load limit, no new worker will be started until the current load is below the limit. No limit will be used when set to 0. Default: no limit

load_limit1=0

load_limit5

Set a limit based on the 5min load average. See load_limit1 for details. Default: no limit

load_limit5=0

load_limit15

Set a limit based on the 15min load average. See load_limit1 for details. Default: no limit

load_limit15=0

idle-timeout

Time in seconds after which an idling worker exits. This parameter controls how fast your waiting workers will exit if there are no jobs waiting. Set to 0 to disable the idle timeout. Default: 10

idle-timeout=30

max-jobs

Controls the amount of jobs a worker will do before he exits. Use this to control how fast the amount of workers will go down after high load times. Disabled when set to 0. Default: 1000

max-jobs=500

fork_on_exec

Use this option to disable an extra fork for each plugin execution. Disabling this option will reduce the load on the worker host, but may cause trouble with unclean plugins. Default: no

fork_on_exec=no

dupserver

sets the address of gearman job server where duplicated result will be sent to. Can be specified more than once to add more server. Useful for duplicating results for a reporting installation or remote gui.

dupserver=logserver:4730,logserver2:4730

show_error_output

Use this option to show stderr output of plugins too. When set to no, only stdout will be displayed. Default is yes.

show_error_output=yes

timeout_return

Defines the return code for timed out checks. Accepted return codes are 0 (Ok), 1 (Warning), 2 (Critical) and 3 (Unknown) Default: 2

timeout_return=2

dup_results_are_passive

Use this option to set if the duplicate result send to the dupserver will be passive or active. Default is yes (passive).

dup_results_are_passive=yes

debug-result

When enabled, the hostname of the executing worker will be put in front of the plugin output. This may help with debugging your plugin results. Default is off.

debug-result=yes

enable_embedded_perl

When embedded perl has been compiled in, you can use this switch to enable or disable the embedded perl interpreter. See Embedded Perl for details on EPN.

enable_embedded_perl=on

use_embedded_perl_implicitly

Default value used when the perl script does not have a "naemon: +epn" or "naemon: -epn" set. Perl scripts not written for epn support usually fail with epn, so its better to set the default to off.

use_embedded_perl_implicitly=off

use_perl_cache

Cache compiled perl scripts. This makes the worker process a little bit bigger but makes execution of perl scripts even faster. When turned off, Mod-Gearman will still use the embedded perl interpreter, but will not cache the compiled script.

use_perl_cache=on

restrict_path

restrict_path allows you to restrict this worker to only execute plugins from these particular folders. Can be used multiple times to specify more than one folder. Note that when this restriction is active, no shell will be spawned and no shell characters ($&();<>`"'|) are allowed in the command line itself.

restrict_path=/usr/local/plugins/

workaround_rc_25

Duplicate jobs from gearmand result sometimes in exit code 25 of plugins because they are executed twice and get killed because of using the same ressource. Sending results (when exit code is 25 ) will be skipped with this enabled. Only needed if you experience problems with plugins exiting with exit code 25 randomly. Default is off.

workaround_rc_25=off

+-----------------------+--------+-------+-------+---------+
| Name                  | Worker | Avail | Queue | Running |
+-----------------------+--------+-------+-------+---------+
| check_results         | 1      | 1     | 0     | 0       |
| host                  | 3      | 3     | 0     | 0       |
| service               | 3      | 3     | 0     | 0       |
| eventhandler          | 3      | 3     | 0     | 0       |
| servicegroup_jmx4perl | 3      | 3     | 0     | 0       |
| hostgroup_japan       | 3      | 3     | 0     | 0       |
+-----------------------+--------+-------+-------+---------+

  ./configure --enable-embedded-perl otheroptions...

Use the supplied check_gearman to monitor your worker and job server. Worker have a own queue for status requests.

This will send a test job to the given job server and the worker will respond with some statistical data.

Job server can be monitored with:

You can use send_gearman to submit active and passive checks to a gearman job server where they will be processed just like a finished check would do.

After installing strawberry perl, you need to install the PAR::Packer module and run pp:

Or just use the prebuild one from labs.consol.de: send_gearman.exe.

check_multi is a plugin which executes multiple child checks. See more details about the feed_passive mode at: www.my-plugin.de

You can pass such child checks to Naemon via the mod_gearman neb module:

If you want to use only check_multi and no other workers, you can achieve this with the following neb module settings:

Set queue_custom_variable=worker in your Mod-Gearman NEB configuration. Then adjust your Naemon host/service configuration and add the custom variable:

The test hostgroup does not have to exist, it is a virtual queue name which is used by the worker.

Adjust your Mod-Gearman worker configuration and put test in the hostgroups attribute. From then on, the worker will work on all jobs in the hostgroup_test queue.

Mod-Gearman has successfully been tested on the following Gearmand Versions. It is recommended to always use the latest listed version of libgearman.

• libgearman 1.1.19.1

• libgearman 0.33

• libgearman 0.32

• libgearman 0.25

• libgearman 0.23

• libgearman 0.14

Mod-Gearman works perfectly with all versions of the Naemon core.

• Naemon

Mod-Gearman dropped support for Nagios 3 starting with Mod-Gearman 4.x. Use a 3.x release from the archive if you still need that.

Mod-Gearman dropped support for Icinga 1 starting with Mod-Gearman 4.x. Use a 3.x release from the archive if you still need that.