The Perfect Load-Balanced & High-Availability Web Cluster With 2 Servers Running Xen On Ubuntu 8.04 Hardy Heron - Page 7

Want to support HowtoForge? Become a subscriber!
 
Submitted by marchost (Contact Author) (Forums) on Fri, 2008-10-03 08:37. ::

13. Server Monitoring With munin And monit (web1, web2)

In this chapter I will describe how you can monitor your webservers node with munin and monit. Munin produces nifty little graphics about nearly every aspect of your server (load average, memory usage, CPU usage, MySQL throughput, eth0 traffic, etc.) without much configuration, whereas monit checks the availability of services like Apache, MySQL, Postfix and takes the appropriate action such as a restart if it finds a service is not behaving as expected. The combination of the two gives you full monitoring: graphics that let you recognize current or upcoming problems (like "We need a bigger server soon, our load average is increasing rapidly."), and a watchdog that ensures the availability of the monitored services.

Although munin lets you monitor more than one server, we will only discuss the monitoring of the system where it is installed here.

 

13.1 Install And Configure munin

apt-get install munin munin-node

Next, we must edit the munin configuration file /etc/munin/munin.conf.

mv /etc/munin/munin.conf /etc/munin/munin.conf.bak
vi /etc/munin/munin.conf

On web1.example.com

dbdir   /var/lib/munin
htmldir /var/www/example/web/monitoring
logdir  /var/log/munin
rundir  /var/run/munin
tmpldir /etc/munin/templates
[web1.example.com]
    address 127.0.0.1
    use_node_name yes

On web2.example.com

dbdir   /var/lib/munin
htmldir /var/www/example/web/monitoring
logdir  /var/log/munin
rundir  /var/run/munin
tmpldir /etc/munin/templates
[web2.example.com]
    address 127.0.0.1
    use_node_name yes

On web1 AND web2

Next we create the directory /var/www/example/web/monitoring and change its ownership to the user and group munin, otherwise munin cannot place its output in that directory. Then we restart munin:

mkdir -p /var/www/example/web/monitoring
chown munin:munin /var/www/example/web/monitoring
/etc/init.d/munin-node restart

Now it is a good idea to password-protect the directory /var/www/example/web/monitoring unless you want everybody to be able to see every little statistic about your server.

To do this, we create an .htaccess file in /var/www/example/web/monitoring:

vi /var/www/example/web/monitoring/.htaccess

AuthType Basic
AuthName "Members Only"
AuthUserFile /var/www/example/monitoring/.htpasswd
<limit GET PUT POST>
require valid-user
</limit>

Then we must create the password file /var/www/example/.htpasswd. We want to log in with the username admin, so we do this:

htpasswd -c /var/www/example/web/monitoring/.htpasswd admin

Enter a password for admin, and you're done!

Now you can access reports (will take a few minutes to collect data) at these addresses :

http://www.example.com:10001/monitoring for web1.example.com

and

http://www.example.com:20001/monitoring for web2.example.com.

 

13.2 Install And Configure monit

To install monit, we do this:

apt-get install monit

Now we must edit /etc/monit/monitrc. The default /etc/monit/monitrc has lots of examples, and you can find more configuration examples on http://www.tildeslash.com/monit/doc/examples.php. However, in my case I want to monitor proftpd, mysql, apache, and postfix, I want to enable the monit web interface on port 2812, I want a https web interface, I want to log in to the web interface with the username admin and the password test, and I want monit to send email alerts to root@localhost, so my file looks like this:

On web1.example.com

cp /etc/monit/monitrc /etc/monit/monitrc_orig
cat /dev/null > /etc/monit/monitrc
vi /etc/monit/monitrc

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: root@web1.example.com }
set alert admin@example.com
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /var/certs/monit.pem
     allow admin:test
#check process vsftpd with pidfile /var/run/vsftpd/vsftpd.pid
#   start program = "/etc/init.d/vsftpd start"
#   stop program  = "/etc/init.d/vsftpd stop"
#   if failed host 192.168.1.104 port 21 protocol ftp then restart
#   if 5 restarts within 5 cycles then timeout
check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout
check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host 192.168.1.104 port 80  protocol http
      and request "/example/web/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout
check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout
check process named with pidfile /var/lib/named/var/run/bind/run/named.pid
   group bind
   start program = "/etc/init.d/bind9 start"
   stop  program = "/etc/init.d/bind9 stop"
   if failed port 53 then restart
   if 5 restarts within 5 cycles then timeout

On web2.example.com

cp /etc/monit/monitrc /etc/monit/monitrc_orig
cat /dev/null > /etc/monit/monitrc
vi /etc/monit/monitrc

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: root@web2.example.com }
set alert admin@example.com
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /var/certs/monit.pem
     allow admin:test
#check process vsftpd with pidfile /var/run/vsftpd/vsftpd.pid
#   start program = "/etc/init.d/vsftpd start"
#   stop program  = "/etc/init.d/vsftpd stop"
#   if failed host 192.168.1.105 port 21 protocol ftp then restart
#   if 5 restarts within 5 cycles then timeout
check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group database
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout
check process apache with pidfile /var/run/apache2.pid
   group www
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host 192.168.1.105 port 80  protocol http
      and request "/example/web/monit/token" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 80% for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout
check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group mail
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout
check process named with pidfile /var/lib/named/var/run/bind/run/named.pid
   group bind
   start program = "/etc/init.d/bind9 start"
   stop  program = "/etc/init.d/bind9 stop"
   if failed port 53 then restart
   if 5 restarts within 5 cycles then timeout

The configuration file is pretty self-explaining; if you are unsure about an option, take a look at the monit documentation: http://www.tildeslash.com/monit/doc/manual.php

In the apache part of the monit configuration you find this:

if failed host www.example.com port 80 protocol http
      and request "/example/web/monit/token" then restart

which means that monit tries to connect to www.example.com on port 80 and tries to access the file /monit/token which is /var/www/example/web/monit/token because our web site's document root is /var/www/example/web. If monit doesn't succeed it means Apache isn't running, and monit is going to restart it.

On web1 AND web2

Now we must create the file /var/www/example/web/monit/token and write some random string into it:

mkdir /var/www/example/web/monit
echo "hello" > /var/www/example/web/monit/token

Next we create the pem cert (/var/certs/monit.pem) we need for the SSL-encrypted monit web interface:

mkdir /var/certs
cd /var/certs

We need an OpenSSL configuration file to create our certificate. It can look like this:

vi /var/certs/monit.cnf

# create RSA certs - Server
RANDFILE = ./openssl.rnd
[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type
[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = MO
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Monitoria
localityName                    = Locality Name (eg, city)
localityName_default            = Monittown
organizationName                = Organization Name (eg, company)
organizationName_default        = Monit Inc.
organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Dept. of Monitoring Technologies
commonName                      = Common Name (FQDN of your server)
commonName_default              = server.monit.mo
emailAddress                    = Email Address
emailAddress_default            = root@monit.mo
[ cert_type ]
nsCertType = server

Now we create the certificate like this:

openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /var/certs/monit.pem -keyout /var/certs/monit.pem

openssl gendh 512 >> /var/certs/monit.pem

openssl x509 -subject -dates -fingerprint -noout -in /var/certs/monit.pem

chmod 700 /var/certs/monit.pem

Afterwards we edit /etc/default/monit to enable the monit daemon. Change startup to 1 and set CHECK_INTERVALS to the interval in seconds that you would like monit to check your system. I choose 60 (seconds) so my file looks like this:

vi /etc/default/monit

# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Fredrik Steen 
# You must set this variable to for monit to start
startup=1
# To change the intervals which monit should run uncomment
# and change this variable.
CHECK_INTERVALS=60

Finally, we can start monit:

/etc/init.d/monit start

Now point your browser to https://192.168.1.104:2812/ or https://192.168.1.105:2812/, log in with admin and test, and you should see the monit web interface.


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by caveman (not registered) on Fri, 2009-03-27 22:11.

please change AuthUserFile /var/www/example/monitoring/.htpasswdtoAuthUserFile /var/www/example/web/monitoring/.htpasswd