How to setup rsyslog for Centralized Log Management

This tutorial will explain how to setup rsyslog as a centralized log management server. Centralized log management means to collect all sorts of logs from several physical or virtualized servers on one log server to monitor the health and security of the server services. We use rsyslog in this tutorial because it offers high-performance, great security and a modular design. It is also capable of storing logs in numerous database solutions like MySQL, Oracle, Hadoop and others for better consolidation.

1. Preliminary Note

For this tutorial, I am using Oracle Linux 6.4 in the 32bit version. Please note that eventhough the configuration are made under Oracle Linux, the same steps will work on CentOS and Red Hat OS Linux. In this tutorial, we will use 2 servers. The first one will act as the rsyslog server and the other will act as workstation / client server for the rsyslog client. By the end of this tutorial, we will see that once a user logs into the client server, the rsyslog server will log the activity made by him automatically.


2. Rsyslog Installation

For the installation phase, we only require to have the rsyslog packages and its dependencies installed. First, let's confirm our version of Operating System.

[[email protected] ~]# cat /etc/issue
Oracle Linux Server release 6.4
Kernel \r on an \m

[[email protected] ~]# arch
i686

[[email protected] ~]# uname -a
Linux RSYS01 2.6.32-358.el6.i686 #1 SMP Fri Feb 22 13:37:29 PST 2013 i686 i686 i386 GNU/Linux


Next, I will configure a new repository to install the rsyslog packages via yum utility.


[[email protected] ~]# cd /etc/yum.repos.d/
[[email protected] yum.repos.d]# vi rsyslog.repo
[rsyslog-v7-devel]
name=Adiscon Rsyslog v7-devel for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-devel/epel-$releasever/$basearch
enabled=0
gpgcheck=0
protect=1

[rsyslog-v7-stable]
name=Adiscon Rsyslog v7-stable for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
protect=1


[[email protected] yum.repos.d]# yum list rsyslog
Loaded plugins: refresh-packagekit, security
rsyslog-v7-stable | 2.5 kB 00:00
rsyslog-v7-stable/primary_db | 188 kB 00:01
Available Packages
rsyslog.i686 7.6.7-1.el6 rsyslog-v7-stable


Done, now let's start to install the new version of rsyslog. The steps are shown below:


[[email protected] yum.repos.d]# yum install rsyslog -y
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rsyslog.i686 0:7.6.7-1.el6 will be installed
--> Processing Dependency: liblogging-stdlog.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libjson-c.so.2 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgthttp.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgtbase.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgt for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libestr.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Running transaction check
---> Package json-c.i686 0:0.11-3.el6 will be installed
---> Package libestr.i686 0:0.1.9-1.el6 will be installed
---> Package libgt.i686 0:0.3.11-1.el6 will be installed
---> Package liblogging.i686 0:1.0.4-1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================
Package Arch Version Repository Size
===============================================================================================================
Installing:
rsyslog i686 7.6.7-1.el6 rsyslog-v7-stable 920 k
Installing for dependencies:
json-c i686 0.11-3.el6 rsyslog-v7-stable 46 k
libestr i686 0.1.9-1.el6 rsyslog-v7-stable 9.0 k
libgt i686 0.3.11-1.el6 rsyslog-v7-stable 55 k
liblogging i686 1.0.4-1.el6 rsyslog-v7-stable 23 k
Transaction Summary
===============================================================================================================
Install 5 Package(s)
Total download size: 1.0 M
Installed size: 3.2 M
Downloading Packages:
(1/5): json-c-0.11-3.el6.i686.rpm | 46 kB 00:00
(2/5): libestr-0.1.9-1.el6.i686.rpm | 9.0 kB 00:00
(3/5): libgt-0.3.11-1.el6.i686.rpm | 55 kB 00:00
(4/5): liblogging-1.0.4-1.el6.i686.rpm | 23 kB 00:00
(5/5): rsyslog-7.6.7-1.el6.i686.rpm | 920 kB 00:03
---------------------------------------------------------------------------------------------------------------
Total 114 kB/s | 1.0 MB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libgt-0.3.11-1.el6.i686 1/5
Installing : liblogging-1.0.4-1.el6.i686 2/5
Installing : libestr-0.1.9-1.el6.i686 3/5
Installing : json-c-0.11-3.el6.i686 4/5
Installing : rsyslog-7.6.7-1.el6.i686 5/5
Verifying : json-c-0.11-3.el6.i686 1/5
Verifying : libestr-0.1.9-1.el6.i686 2/5
Verifying : liblogging-1.0.4-1.el6.i686 3/5
Verifying : libgt-0.3.11-1.el6.i686 4/5
Verifying : rsyslog-7.6.7-1.el6.i686 5/5

Installed:
rsyslog.i686 0:7.6.7-1.el6

Dependency Installed:
json-c.i686 0:0.11-3.el6 libestr.i686 0:0.1.9-1.el6 libgt.i686 0:0.3.11-1.el6 liblogging.i686 0:1.0.4-1.el6

Complete!

[[email protected] yum.repos.d]# rsyslogd -v
rsyslogd 7.6.7, compiled with:
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.

[[email protected] ~]# rpm -qa|grep rsyslog
rsyslog-7.6.7-1.el6.i686


Next, let's proceed with the installation phase. For this tutorial, we will skip on the security layer to make things simple. We will disable SELINUX to ensure that there's no issue related security raise during the way. Be aware that you should not disable SELinux and the Firewall on a live setup. Below are the steps:

First, check what is the current status for our SELINUX policy.

[[email protected] ~]# getenforce
Enforcing

To disable it permenantly, follow the step below:

[[email protected] ~]# cd /etc/sysconfig/
[[email protected] ~]# vi selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted


After that let's ensure we have disabled the firewall to avoid any blocking between server and client connection.

[[email protected] ~]# iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[[email protected] ~]# /etc/init.d/iptables stop
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Unloading modules: [ OK ]

[[email protected] ~]# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination


Done, now as the installation phase has succeed. Let's move on to configuration phase.


3. Rsyslog Configuration

All package dependencies has been installed already, let's focus on the configuration of rsyslog. Let's go inside the configuration file and make the changes like below:

[[email protected] ~]# vi /etc/rsyslog.conf

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

$template Auditlog, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.* ?Auditlog

Below is the explaination on the configuration changes we made:

  • module(load="imudp") # needs to be done just once ==> We will provide ability to receive rsyslog message between server and client via UDP protocol
  • input(type="imudp" port="514") ==> We will use port 514 for rsyslog services
  • $template Auditlog, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log" ==> For every client connected to rsyslog server, system will automatically create the client hostname folder and related services file name
  • $template TmplMsg, "/var/log/rsyslog_client/%HOSTNAME%/%PROGRAMNAME%.log" ==> For every client connected to rsyslog server, system will automatically create the client hostname folder and related services file name


Once the configuration is done, let's start the rsyslog service.

[[email protected] yum.repos.d]# /etc/init.d/rsyslog restart
Shutting down system logger: [FAILED]
Starting system logger: [ OK ]

Use netstat o ensure whether the rsyslog services are up and running:

[[email protected] yum.repos.d]# netstat -uanp|grep rsyslog
udp 0 0 0.0.0.0:514 0.0.0.0:* 2430/rsyslogd
udp 0 0 :::514 :::* 2430/rsyslogd

Above you can see that our rsyslog service is running with the port assignements we have made. By default, rsyslog also audit its own server access as it treats the rsyslog server as a client. To ensure whether it's working, we can take a look into the /var/log folder. Let's check whether a folder with the name RSYS01 (that's the server hostname) exist or not.

[[email protected] log]# cd /var/log/
[[email protected] log]# ls -l|grep rsyslog
drwx------ 3 root root 4096 Oct 24 18:21 rsyslog_client

[[email protected] log]# cd rsyslog_client
[[email protected] rsyslog_client]# ls
RSYS01
[[email protected] rsyslog_client]# cd RSYS01/
[[email protected] RSYS01]# ls
rsyslogd.log

Nice, seems everything works like a charm! Now let's proceed with the testing phase to conclude all configurations are made as expected.

4. Testing Phase

As the server that runs the rsyslog service also works as a client, we can see on the server itself if monitoring of local logins is monitored. To ensure whether it is true, let's log into the rsyslog server with another session by using the SSH service. For this step, we will assume that the server itself already configured passwordless configuration. Below are the steps:

[[email protected] RSYS01]# ssh [email protected]
Last login: Sat Oct 22 15:45:48 2016 from 172.20.181.70

[[email protected] ~]# who
root pts/0 2016-10-22 00:21 (172.20.181.11)
root pts/1 2016-10-24 18:22 (127.0.0.1)

[[email protected] ~]# exit
logout
Connection to RSYS01 closed.

Done, as simple as that. We just log into the rsyslog server itself then once the new session is created, we simply log again just to ensure the rsyslog service did audit the session. Now, let's check whether the session has been audited or not. Below are the steps:

[[email protected] ~]# cd var/log/rsyslog_client/RSYS01
[[email protected] RSYS01]# ls
rsyslogd.log sshd.log
[[email protected] RSYS01]# tail -f sshd.log
Oct 24 18:22:46 RSYS01 sshd[2536]: Accepted password for root from 192.168.43.101 port 52862 ssh2
Oct 24 18:22:46 RSYS01 sshd[2536]: pam_unix(sshd:session): session opened for user root by (uid=0)
Oct 24 18:22:50 RSYS01 sshd[2536]: Received disconnect from 192.168.43.101: 11: disconnected by user
Oct 24 18:22:50 RSYS01 sshd[2536]: pam_unix(sshd:session): session closed for user root
^C


Excellent, the rsyslog service automatically created a sshd.log file when the session started on the server. Inside the log file, we can see that there's list of detail stated the time, port and user created under the session.

Now as everything works as expected. Let's setup a workstation for the rsyslog client to be audited by our rsyslog server. For the rsyslog client, you just need to install the rsyslog packages and make a simple change in the configuration file to link to the rsyslog server. Below are the steps:

[[email protected] ~]# cd /etc/yum.repos.d/
[[email protected] yum.repos.d]# yum list rsyslog
Loaded plugins: refresh-packagekit, security
rsyslog-v7-stable | 2.5 kB 00:00
rsyslog-v7-stable/primary_db | 188 kB 00:01
Available Packages
rsyslog.i686 7.6.7-1.el6 rsyslog-v7-stable


[[email protected] yum.repos.d]# yum install rsyslog -y
Loaded plugins: refresh-packagekit, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package rsyslog.i686 0:7.6.7-1.el6 will be installed
--> Processing Dependency: liblogging-stdlog.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libjson-c.so.2 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgthttp.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgtbase.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libgt for package: rsyslog-7.6.7-1.el6.i686
--> Processing Dependency: libestr.so.0 for package: rsyslog-7.6.7-1.el6.i686
--> Running transaction check
---> Package json-c.i686 0:0.11-3.el6 will be installed
---> Package libestr.i686 0:0.1.9-1.el6 will be installed
---> Package libgt.i686 0:0.3.11-1.el6 will be installed
---> Package liblogging.i686 0:1.0.4-1.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================
Package Arch Version Repository Size
===============================================================================================================
Installing:
rsyslog i686 7.6.7-1.el6 rsyslog-v7-stable 920 k
Installing for dependencies:
json-c i686 0.11-3.el6 rsyslog-v7-stable 46 k
libestr i686 0.1.9-1.el6 rsyslog-v7-stable 9.0 k
libgt i686 0.3.11-1.el6 rsyslog-v7-stable 55 k
liblogging i686 1.0.4-1.el6 rsyslog-v7-stable 23 k

Transaction Summary
===============================================================================================================
Install 5 Package(s)

Total download size: 1.0 M
Installed size: 3.2 M
Downloading Packages:
(1/5): json-c-0.11-3.el6.i686.rpm | 46 kB 00:00
(2/5): libestr-0.1.9-1.el6.i686.rpm | 9.0 kB 00:00
(3/5): libgt-0.3.11-1.el6.i686.rpm | 55 kB 00:00
(4/5): liblogging-1.0.4-1.el6.i686.rpm | 23 kB 00:00
(5/5): rsyslog-7.6.7-1.el6.i686.rpm | 920 kB 00:03
---------------------------------------------------------------------------------------------------------------
Total 114 kB/s | 1.0 MB 00:09
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : libgt-0.3.11-1.el6.i686 1/5
Installing : liblogging-1.0.4-1.el6.i686 2/5
Installing : libestr-0.1.9-1.el6.i686 3/5
Installing : json-c-0.11-3.el6.i686 4/5
Installing : rsyslog-7.6.7-1.el6.i686 5/5
Verifying : json-c-0.11-3.el6.i686 1/5
Verifying : libestr-0.1.9-1.el6.i686 2/5
Verifying : liblogging-1.0.4-1.el6.i686 3/5
Verifying : libgt-0.3.11-1.el6.i686 4/5
Verifying : rsyslog-7.6.7-1.el6.i686 5/5

Installed:
rsyslog.i686 0:7.6.7-1.el6

Dependency Installed:
json-c.i686 0:0.11-3.el6 libestr.i686 0:0.1.9-1.el6 libgt.i686 0:0.3.11-1.el6 liblogging.i686 0:1.0.4-1.el6

Complete!


Done, now we've installed the rsyslog package on our client workstation. Now let's make an amendment inside the rsyslog configuration file. For a client configuration, you just need to amend the configuration like below:

[[email protected] ~]# vi /etc/rsyslog.conf

*.* @192.168.43.101:514

That's all, noted that we have included IP 192.168.43.101 with port 514 in the configuration file. That IP is the IP for rsyslog server. Now as everything is done, let's restart the rsyslog service on the client workstation to load the changes. Below are the steps:

[[email protected] ~]# /etc/init.d/rsyslog restart
Shutting down system logger: [FAILED]
Starting system logger: [ OK ]


Now let's go back to our rsyslog server and see if a folder for the rsyslog client hostname has been created in the rsyslog log directory. Below are the steps:

[[email protected] ~]# cd var/log/rsyslog_client/
[[email protected] rsyslog_client]# ls
RSYS01 CLIENT01


Excellent, note that a folder with rsyslog client hostname has been created automatically. This confirms that our configuration is correct and the rsyslog client is able to make a UDP connection to the rsyslog server.

For the next test procedure, let's log into rsyslog client as another user and see if the rsyslog server manages to capture the activity or not. Below are the steps:

::CLIENT01::
login as: shahril
[email protected]'s password:
Last login: Sun Oct 23 00:21:40 2016 from 172.20.181.11

[[email protected] ~]$ who
shahril pts/0 2016-10-24 17:01 (192.168.43.80)

[[email protected] ~]$ exit


Now, let's check the log directory inside the rsyslog server to see if we managed to log the activity created from rsyslog client or not.

[[email protected] ~]# cd var/log/rsyslog_client/
[[email protected] rsyslog_client]# cd CLIENT01/
[[email protected] CLIENT01]# ls
rsyslogd.log sshd.log
[[email protected] CLIENT01]# tail -10 sshd.log
Oct 24 17:01:47 CLIENT01 sshd[2102]: Accepted password for shahril from 192.168.43.80 port 17002 ssh2
Oct 24 17:01:47 CLIENT01 sshd[2102]: pam_unix(sshd:session): session opened for user shahril by (uid=0)


Great, result shows the process works as expected. Now for the final testing, let's log back into the rsyslog client and install a package to test if the rsyslog service manages to log other activity than session creation. Below are the steps:

::CLIENT01::
login as: root
[email protected]'s password:
Last login: Sat Oct 22 10:21:40 2016 from 172.20.181.11

[[email protected] ~]# yum install firefox -y
Loaded plugins: refresh-packagekit, security
Repository 'OEL64' is missing name in configuration, using id
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package firefox.i686 0:10.0.12-1.0.1.el6_3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
firefox i686 10.0.12-1.0.1.el6_3 OEL64 20 M

Transaction Summary
================================================================================
Install 1 Package(s)

Total download size: 20 M
Installed size: 23 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : firefox-10.0.12-1.0.1.el6_3.i686 1/1
Verifying : firefox-10.0.12-1.0.1.el6_3.i686 1/1

Installed:
firefox.i686 0:10.0.12-1.0.1.el6_3

Complete!

The above shows that we have successfully installed a firefox browser installer inside our rsyslog client workstation. Now let's go back to our rsyslog server and check if rsyslog was able to log the process of installing a third party package into the workstation. Below are the steps:

[[email protected] ~]# cd var/log/rsyslog_client/
[[email protected] CLIENT01]# ls
rsyslogd.log sshd.log yum.log
[[email protected] CLIENT01]# tail -20 yum.log
Oct 25 17:13:17 CLIENT01 yum[2319]: Installed: firefox-10.0.12-1.0.1.el6_3.i686

Excellent, rsyslog service managed to audit the information of the installation activity in the client workstation.

Share this page:

Suggested articles

5 Comment(s)

Add comment

Comments

From: Jukka

You might want to switch to using relp protocol with tls certificates so you won't be sending your logs in cleartext over udp connections:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html

http://www.rsyslog.com/doc/v8-stable/tutorials/tls_cert_summary.html

Disabling SELinux is neither necessary for this as all you might have to do is add the port being used to syslog_tls_port_t type.

From: Sado

Are you going to start using SELinux and firewall rules at all?

From: chris riney

I agree, you should be moving away from 514/UDP to syslog-tls (6514/tcp), or atleast syslog-conn(601/tcp), and be configuring the reliable delivery of the syslog data.

Also why are you not using the rsyslog package provided with the Oracle/CentOS/RedHat base distribution, other than it may be slightly out of date?

The rsyslog package should already have the proper rules for all the possible valid ports configured for SELinux.

From: shahril bin kamaruzzaman

Great suggestion guys !

 

As for this exercise I use EL 6.4 therefore the bundled rsyslog package are 2 version late than I used on above. 

From: Pete Vargas-Mas

I've always had a problem with people who disable SELinux. A security package is there for a reason. Instead of disabling it, we should be learning how to make things work with it enforcing. By they way, if you take a Red Hat Certification Exam, SELinux must be enabled in order to pass the test.