4 Adding A Remote Server (server2) To Icinga

Monitoring localhost is nice, but of course, it would be even better if we could monitor all of our servers in one location. This is possible with Icinga, and this chapter describes how we can add our second Ubuntu 11.10 server (server2.example.com) to the setup.

To do this, we need to install the Nagios NRPE (Nagios Remote Plugin Executor) server on server2, and the Nagios NRPE plugin on server1. The NRPE server will listen on server2; server1 will connect to it using the NRPE plugin and pass commands to it that the NRPE server will execute on server2; it will pass back the results to server1.

First we install the nagios-nrpe-plugin package on server1:

server1:

apt-get install nagios-nrpe-plugin

Nagios web administration password: <-- nagiosadmin_password
Password confirmation: <-- nagiosadmin_password

Now we go to server2:

server2:

Install the nagios-nrpe-server package:

apt-get install nagios-nrpe-server

Now open /etc/nagios/nrpe.cfg:

vi /etc/nagios/nrpe.cfg

We must configure the NRPE server to allow server1 (IP: 192.168.0.100) to connect, therefore we add 192.168.0.100 to the allowed_hosts line:

[...]
# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address.  I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1,192.168.0.100
[...]

(If you don't do this, you will get the following error when you run

/usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101

on server1:

root@server1:/etc/nagios-plugins/config# /usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101
CHECK_NRPE: Error - Could not complete SSL handshake.
root@server1:/etc/nagios-plugins/config#

)

Also, server1 needs to be allowed to pass command line arguments to the NRPE server, so still in the same file we set dont_blame_nrpe to 1:

[...]
# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed.  This option only works
# if the daemon was configured with the --enable-command-args configure script
# option.
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! ***
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=1
[...]

(If you don't do this, you will see the error

CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.

for lots of remote service checks in the Icinga web interface, and in /var/log/syslog on server2 you will see these errors:

Aug 23 14:20:20 server2 nrpe[11496]: Error: Request contained command arguments, but argument option is not enabled!
Aug 23 14:20:20 server2 nrpe[11496]: Client request was invalid, bailing out...

)

Finally we must add command definitions for each service check we want to run on server2 and that is not already defined. I want to run the the check_procs, check_all_disks, and check_mysql_cmdlinecred checks on server2; these are not defined in /etc/nagios/nrpe.cfg, so I add them now (I also want to run the check_users and check_load checks, but these are already defined):

[...]
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w 250 -c 400
command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w '20%' -c '10%' -e
command[check_mysql_cmdlinecred]=/usr/lib/nagios/plugins/check_mysql -H localhost -u 'nagios' -p 'howtoforge'
[...]

(If you don't do this, you will get errors like

NRPE: Command 'check_all_disks' not defined
NRPE: Command 'check_mysql_cmdlinecred' not defined
NRPE: Command 'check_procs' not defined

in the Icinga web interface.)

As you see I have hardcoded the command line arguments because using variables like command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ did not work for me. But still, when we configure the service checks for server2 on server1, we will have to pass command line arguments to these checks; server2 will ignore these because I have hardcoded the comand line arguments into /etc/nagios/nrpe.cfg, but if you leave them out, you will get errors like /usr/lib/nagios/plugins/check_nrpe: option requires an argument -- 'a' in the Icinga web interface.

Now save the file and restart the NRPE server:

/etc/init.d/nagios-nrpe-server restart

Now check if the NRPE server is listening:

netstat -tap | grep nrpe

root@server2:~# netstat -tap | grep nrpe
tcp        0      0 *:nrpe                  *:*                     LISTEN       23668/nrpe
root@server2:~#

Now go back to server1...

server1:

... and check if it can connect to the NRPE server on server2:

/usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101

Output should be as follows in case of success:

root@server1:~# /usr/lib/nagios/plugins/check_nrpe -H 192.168.0.101
NRPE v2.12
root@server1:~#

Now go back to server2:

server2:

We want to check MySQL on server2; because we use the NRPE daemon, we can run the check locally on server2, i.e., we don't have to open MySQL to the outside to allow server1 to run the check. Therefore I create the MySQL user nagios for localhost and localhost.localdomain instead of for 192.168.0.100 and server1.example.com:

mysql -u root -p

GRANT USAGE ON *.* TO nagios@localhost IDENTIFIED BY 'howtoforge';
GRANT USAGE ON *.* TO nagios@localhost.localdomain IDENTIFIED BY 'howtoforge';
FLUSH PRIVILEGES;

quit;

Now we go back to server1...

server1:

... and create the Icinga configuration for server2:

vi /etc/icinga/objects/server2_icinga.cfg

define host{
       use generic-host
       host_name server2.example.com
       alias server2
       address 192.168.0.101
}
define service{
       use generic-service
       host_name server2.example.com
       service_description PING
       check_command check_ping!100.0,20%!500.0,60%
}
define service{
       use                             generic-service         ; Name of service template to use
       host_name                       server2.example.com
       service_description             Disk Space
       check_command                   check_nrpe!check_all_disks!20%!10%
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             Current Users
       check_command                   check_nrpe!check_users!20!50
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             Total Processes
       check_command                   check_nrpe!check_procs!250!400
}
define service{
        use                             generic-service         ; Name of service template to use
        host_name                       server2.example.com
        service_description             Current Load
        check_command                   check_nrpe!check_load!5.0!4.0!3.0!10.0!6.0!4.0
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             MySQL
       check_command                   check_nrpe!check_mysql_cmdlinecred!nagios!howtoforge
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             SMTP
       check_command                   check_smtp
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             POP3
       check_command                   check_pop
}
define service{
       use                             generic-service
       host_name                       server2.example.com
       service_description             IMAP
       check_command                   check_imap
}

(As I've mentioned before, although I have hardcoded the command line arguments for some commands into /etc/nagios/nrpe.cfg on server2, we still need to add command line arguments to certain these checks here.)

As you see, I use check_nrpe for some checks and pass the actual check (like check_all_disks) as a command line argument to check_nrpe. These are the checks that will be executed locally by the NRPE server on server2. check_nrpe is not needed for all checks. Checks that test a connection from the outside like check_ping or check_smtp can be run from server1.

To check the SSH and HTTP services on server2, we can EITHER add the following stanzas to /etc/icinga/objects/server2_icinga.cfg...

[...]
define service {
        use                             generic-service
        host_name                       server2.example.com
        service_description             SSH
        check_command                   check_ssh
}
define service {
        use                             generic-service
        host_name                       server2.example.com
        service_description             HTTP
        check_command                   check_http
}

... OR we add server2.example.com to the http-servers and ssh-servers hostgroups in /etc/icinga/objects/hostgroups_icinga.cfg:

vi /etc/icinga/objects/hostgroups_icinga.cfg

# Some generic hostgroup definitions

# A simple wildcard hostgroup
define hostgroup {
        hostgroup_name  all
                alias           All Servers
                members         *
        }

# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
                alias           Debian GNU/Linux Servers
                members         localhost,server2.example.com
        }

# A list of your web servers
define hostgroup {
        hostgroup_name  http-servers
                alias           HTTP servers
                members         localhost,server2.example.com
        }

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
                alias           SSH servers
                members         localhost,server2.example.com
        }

Restart Icinga:

/etc/init.d/icinga restart

Afterwards you should find server2 in the Icinga web interface:

Here are the service checks for server2:

If you have added server2 to the hostgroups, it should be listed under Service Overview For All Host Groups as well:

5 Links

• Icinga: https://www.icinga.org/
• Icinga Documentation: http://docs.icinga.org/latest/en/
• Nagios: http://www.nagios.org/
• Ubuntu: http://www.ubuntu.com/