Enhanced Logging With rsyslog On Debian Etch And phpLogcon For Viewing

Want to support HowtoForge? Become a subscriber!
 
Submitted by Miguel (Contact Author) (Forums) on Mon, 2007-10-01 13:15. :: Debian | Monitoring

Enhanced Logging With rsyslog On Debian Etch And phpLogcon For Viewing

Well everybody knows the issues of reading and searching in log files. If you have more than one machine it even gets worse. This tutorial describes how to install and configure rsyslog on Debian Etch, but it can be adapted to other distributions.

Exerpt from rsyslog site:

"Rsyslog, the enhanced syslogd for Linux and Unix.

Rsyslog is an enhanced multi-threaded syslogd supporting, among others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, filtering on any message part, and fine grain output format control. It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its advanced features make it suitable for enterprise-class, encryption protected syslog relay chains while at the same time being very easy to setup for the novice user. An optional web interface - phpLogCon - can be used to visualize all data online."

In this tutorial we will be building rsyslog from source and write the necessary config files.

 

Step 1: See That The Necessary Tools Are Installed

apt-get install binutils cpp fetchmail flex gcc libarchive-zip-perl libc6-dev libcompress-zlib-perl libdb4.3-dev libpcre3 libpopt-dev linux-kernel-headers lynx m4 make ncftp nmap openssl perl perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++ mysql-server mysql-client libmysqlclient15-dev

Next set a password for your mysql root user:

mysqladmin -u root password your_mysqlroot_password 

Now we can create the rsyslog db:

mysqladmin -u root -p create rsyslog

Next we start the mysql command shell and create the rsyslog user:

mysql -u root -p
GRANT SELECT, INSERT, UPDATE, DELETE ON rsyslog.* TO 'rsyslog_user'@'localhost' IDENTIFIED BY 'rsyslog_user_password';
GRANT SELECT, INSERT, UPDATE, DELETE ON mail.* TO 'rsyslog_user'@'localhost.localdomain' IDENTIFIED BY 'rsyslog_user_password';
FLUSH PRIVILEGES;
quit 

As you might have noticed we do not import any tables into the db. phpLogCon will do that for us. 

 

Step 2: Get The rsyslog Source And Build rsyslog

Get the source:

cd /tmp
wget http://www.rsyslog.com/Downloads-req-getit-lid-58.phtml 

Now let's build and install rsyslog:

tar xvzf rsyslog-1.9.6.tar.gz
cd rsyslog-1.9.6
./configure
make
make install 

Since rsyslog has no configuration examples (and files) I'll provide the examples here. All of these are just default configurations, use the documentation provided by rsyslog to customize the configuration to your specific needs.

First the rsyslog configuration file /etc/rsyslog.conf

vi /etc/rsyslog.conf

Just copy and paste the contents below:

Do not forget to change the line:

*.*       >127.0.0.1,rsyslog,rsyslog_user,rsyslog_user_password

with the username and password you specified above.

#  /etc/rsyslog.conf    Configuration file for rsyslogd.
#
#                       For more information see
#                       /usr/share/doc/rsyslog/html/rsyslog_conf.html
#
# First some standard logfiles.  Log by facility.
#
$ModLoad MySQL
*.*       >127.0.0.1,rsyslog,rsyslog_user,rsyslog_user_password
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log
kern.*                          -/var/log/kern.log
lpr.*                           -/var/log/lpr.log
mail.*                          -/var/log/mail.log
user.*                          -/var/log/user.log
#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info                       -/var/log/mail.info
mail.warn                       -/var/log/mail.warn
mail.err                        /var/log/mail.err
#
# Logging for INN news system
#
news.crit                       /var/log/news/news.crit
news.err                        /var/log/news/news.err
news.notice                     -/var/log/news/news.notice
#
# Some `catch-all' logfiles.
#
*.=debug;\
        auth,authpriv.none;\
        news.none;mail.none     -/var/log/debug
*.=info;*.=notice;*.=warn;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg                         *
#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#       news.=crit;news.=err;news.=notice;\
#       *.=debug;*.=info;\
#       *.=notice;*.=warn       /dev/tty8
# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
# you must invoke `xconsole' with the `-file' option:
#
#    $ xconsole -file /dev/xconsole [...]
#
# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
#      busy site..
#
daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
        *.=notice;*.=warn       |/dev/xconsole
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/

Next is the startup script /etc/init.d/rsyslog

vi /etc/init.d/rsyslog

Just copy and paste the contents below:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          syslog
# Required-Start:    $local_fs $time
# Required-Stop:     $local_fs $time
# Should-Start:      $network
# Should-Stop:       $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enhanced syslogd
# Description:       Rsyslog is an enhanced multi-threaded syslogd.
#                    It is quite compatible to stock sysklogd and can be
#                    used as a drop-in replacement.
### END INIT INFO
# Author: Michael Biebl <biebl@debian.org>
#
# Do NOT "set -e"
# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="enhanced syslogd"
NAME=rsyslog
RSYSLOGD=rsyslogd
RSYSLOGD_BIN=/usr/local/sbin/rsyslogd
RSYSLOGD_OPTIONS="-m 0"
RSYSLOGD_PIDFILE=/var/run/rsyslogd.pid
RKLOGD=rklogd
RKLOGD_BIN=/usr/local/sbin/rklogd
RKLOGD_OPTIONS="-2"
RKLOGD_PIDFILE=/var/run/rklogd.pid
SCRIPTNAME=/etc/init.d/$NAME
# Exit if the package is not installed
[ -x "$RSYSLOGD_BIN" ] || exit 0
[ -x "$RKLOGD_BIN" ] || exit 0
# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions
#
# Function that starts the daemon/service
#
do_start()
{
        DAEMON=$1
        DAEMON_ARGS=$2
        PIDFILE=$3
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
                || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
}
#
# Function that stops the daemon/service
#
do_stop()
{
        NAME=$1
        PIDFILE=$2
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        #rm -f $PIDFILE
        return "$RETVAL"
}
#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        NAME=$1
        PIDFILE=$2
        start-stop-daemon --stop --signal HUP --quiet --pidfile $PIDFILE --name $NAME
        return 0
}
create_xconsole() {
        if [ ! -e /dev/xconsole ]
        then
                mknod -m 640 /dev/xconsole p
        fi
}
case "$1" in
  start)
        log_daemon_msg "Starting $DESC" "$RSYSLOGD"
        create_xconsole
        do_start "$RSYSLOGD_BIN" "$RSYSLOGD_OPTIONS" "$RSYSLOGD_PIDFILE"
        case "$?" in
#               0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        log_progress_msg "$RKLOGD"
        do_start "$RKLOGD_BIN" "$RKLOGD_OPTIONS" "$RKLOGD_PIDFILE"
        case "$?" in
                0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        ;;
  stop)
        log_daemon_msg "Stopping $DESC" "$RKLOGD"
        do_stop "$RKLOGD" "$RKLOGD_PIDFILE"
        case "$?" in
#               0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        log_progress_msg "$RSYSLOGD"
        do_stop "$RSYSLOGD" "$RSYSLOGD_PIDFILE"
        case "$?" in
                0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        ;;
  reload|force-reload)
        log_daemon_msg "Reloading $DESC" "$RSYSLOGD"
        do_reload "$RSYSLOGD" "$RSYSLOGD_PIDFILE"
        log_end_msg $?
        ;;
  restart)
        $0 stop
        $0 start
        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        exit 3
        ;;
esac
:

Next we create a default configuration in /etc/default/rsyslogd

vi /etc/default/rsyslog

Just copy and paste the contents below:

# Options to rsyslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS="-m 0"
# Options to rklogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
#    once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See rklogd(8) for more details
RKLOGD_OPTIONS="-x"

Next we create the directory /etc/rsyslog.d - this is needed for additional configuration options (not covered in the tutorial). 

mkdir /etc/rsyslog.d

Now we have all of the configuration files, but we need now to make sure rsyslog is started at boot time in the right moment. Please note that rsyslog is not compatible with klogd and sysklogd which are installed by default. These need to be disabled or de-installed. In this tutorial I will disable them.

Issue the following commands to start rsyslogd at the right time during system boot and to disable klogd and sysklogd.

ln -s  /etc/init.d/rsyslog /etc/rc3.d/S10rsyslog
mv /etc/rc3.d/S10sysklogd /etc/rc3.d/_S10sysklogd
mv /etc/rc3.d/S11klogd /etc/rc3.d/_S11klogd

This concludes the build and configuration part.

 

Step 3: Install Apache2 And PHP5

You can skip this if you have already installed apache2 and php5. 

apt-get install apache2 apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert

apt-get install libapache2-mod-php5 php5 php5-common php5-curl php5-dev php5-gd php5-idn php-pear php5-imagick php5-imap php5-json php5-mcrypt php5-memcache php5-mhash php5-ming php5-mysql php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl 

You will be asked the following question:

Continue installing libc-client without Maildir support? <-- Yes

 

Step 4: Get phpLogCon And Install phpLogCon

cd /tmp
wget  http://www.phplogcon.org/Downloads-req-getit-lid-6.phtml

Next we will create a directory for phpLogCon and enable it in Apache2:

tar xvzf phplogcon-1.2.3.tar.gz
mkdir /var/www/phplogcon
cp -R phplogcon-1.2.3/* /var/www/phplogcon 

Configure apache2 for phpLogcon:

vi /etc/apache2/sites-enabled/your_site_conf 

Add the following alias to enable phplogcon: 

Alias /phplogcon "/var/www/phplogcon"

Insert the line above before </virtualhost> or if you have already aliases defined, insert the line there.

Note: if /var/www is your server root then you don't need to create an alias. 

Restart apache2 to enable the changes.

/etc/init.d/apache2 force-reload 

 

Step 5: Configure phpLogCon And Install The Tables In The rsyslog DB

Point your browser to http://yourdomain.tld/phplogcon and follow the steps on the screen. After the installation wizard has finished, the database is populated with all of the required tables and an administrative user is created.

Delete the install directory to enable phpLogCon:

rm -R /var/www/phplogcon/install

 

Step 6: Start rsyslog

Issue the following commands to start rsyslog and to stop klogd and sysklogd:

/etc/init.d/sysklogd stop
/etc/init.d/klogd stop
/etc/init.d/rsyslog start

 

Step 7: Enjoy rsyslog

Point your browser to http://yourdomain.tld/phplogcon and log in with the username and password you specified in the installation wizard. You should now see something like this:

phpLogCon Screenshot


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by BigThetan (registered user) on Thu, 2010-05-27 19:13.

Thanks for this howto. I got confused on step 4 were it says:

 Configure apache2 for phpLogcon:


vi /etc/apache2/sites-enabled/your_site_conf

 I dont have a Webiste  so what do I put where it says 'your_site_conf'?

 Thanks in Advance

Submitted by Steven Martin (not registered) on Wed, 2008-09-10 20:40.
As well as updating the WGET url I had to use ./configure --enable-mysql when installing 3.18.3 (stable)
Submitted by mac86 (registered user) on Wed, 2008-03-05 19:17.
very nice your article and phplogCon.
Submitted by Rainer Gerhards (not registered) on Fri, 2008-09-12 16:02.
Great article! I'd just like to add that there is a new, much more feature rich version of phpLogCon out. It is well worth looking at it.
Submitted by Gorkem Cetin (not registered) on Mon, 2008-10-13 01:08.

I tried the final version of phpLogCon (2.5.12). I liked the design, its solution based approach and rich feature set. One thing I'd love to see in future versions is usability and intuitiveness of setup procedure.

Thanks!