Client Setup

Log in on the server you wish to install the Samhain client on and make sure you are the root user. Also make sure you have all the essential build packages installed, refer to the overview for installation of these essential build packages.

First, we need a gpg key for root.

See previous example for detailed steps.

gpg --gen-key

Now we need to pull out the fingerprint for this key, so that we can use it when building the Samhain binary.

MY_FP=`gpg --fingerprint root | grep fingerpr | sed 's/ //g' | awk 'BEGIN { FS = "=" } ; {print $2}'`

As before, we need TIGER192 checksum capability first.

./configure
make

Now, since we are having a bit of fun, we are going to change the name of the binary and process. Classical security by obscurity. I'm picking the name john, a general purpose password cracker. Pick a name that will not stand out in a process listing and shout out "THIS IS A HIDS PROCESS !!!!11". Then again, know how much/little a name change actually hides what this binary does before you rely on it to hide you're HIDS from a l33t haxor.

We further specify that the configuration and data files should be pulled from the server. If you want to take this one step further, look into the following compile options : --enable-khide,--enable-suidcheck and --with-kcheck=/path/to/System.map

Make sure to change IP_OF_YOUR_SERVER to the actual IP address of your Yule server.

./configure --with-gpg=/usr/bin/gpg --enable-network=client --with-config-file=REQ_FROM_SERVER --with-data-file=REQ_FROM_SERVER/var/lib/john/john \
--enable-stealth=129 --enable-install-name=john --enable-srp --with-fp=$MY_FP --with-port=50888 \
--with-logserver=IP_OF_YOUR_SERVER --with-sender=john
make

Make the required directories, copy the binary over (with the correct name) and put the initialization script in place.

mkdir /var/lib/john/
cp init/samhain.startLinux /etc/init.d/john
chmod 744 /etc/init.d/john
cp samhain /usr/local/sbin/john
cp samhain_setpwd /usr/local/sbin/john_setpwd
cp samhain_stealth /usr/local/sbin/john_stealth
cd /usr/local/sbin

Set the password and overwrite the binary.

/usr/local/sbin/john_setpwd john jingle 161718abcd212324
mv john.jingle john

"jingle" Does not matter, it's just the append and the number is what you want in 16 bit 0-9, A-F (A.K.A HEX). You can use yule -G on the server to generate a random number for you.

The output should look something like:

INFO old password found
INFO replaced: f7c312aaaa12c3f7 by: 161718abcd212324
INFO finished

Change the description in the initialization script.

sed -i 's/File Integrity Checking/Password Cracking/' /etc/init.d/john

Make sure the daemon starts at boot.

Red Hat

chkconfig --add john
chkconfig john on

Debian

update-rc.d john defaults

A Little Work On The Server

The HEX key we just embedded in the client binary, we need it now to tell the server about that client.

/usr/local/sbin/yule -P 161718abcd212324 | sed 's/HOSTNAME/CliENT_HOSTNAME_HERE/' >> /etc/yulerc <-- Make sure to put the client host name (FQDN) in.

Edit /etc/yulerc and move the key above the GPG signature.

For example, the last couple of lines of /etc/yulerc mihgt look like this:

# Client=HOSTNAME@00000000@C39F0EEFBC64E4A8BBF72349637CC07577F714B420B62882
# Client=HOSTNAME@8F81BA58956F8F42@8932D08C49CA76BD843C51EDD1D6640510FA032A7A2403E572BBDA2E5C6B753991CF7E091141D20A2499C5CD3E14C1639D17482E14E1548E5246ACF4E7193D524CDDAC9C9D6A9A36C596B4ECC68BEB0C5BB7082224946FC98E3ADE214EA1343E2DA8DF4229D4D8572AD8679228928A787B6E5390D3A713102FFCC9D0B2188C92
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
iD8DBQFNEI7oerU1Wrr7a5ERAkWqAJ9sZEuRLp8rPOjXdUokT03bEfjGuwCfa+Tr
pDK7/KmGj3Hx8vRMufxNx7A=
=zI4S
-----END PGP SIGNATURE-----
Client=rhc2.sys.local@F1DF72033799940C@37FC42534A812B2351007A24820537466495F97ED352EFC1D9DCAEACBBF5CB98AEF183057CE6D101151F112693C2DAE361435CED1C95E822272FE287A56B4D38EE91B00830A56AE2F26E4738DF099CAEF3372342BE0ACDB78C12FD176EED1FBA376A0399537F848B6FA9AD4E61E6C771A5566F43D62C1F9836AB976CB1111545

We need to change that to look like this:

# Client=HOSTNAME@00000000@C39F0EEFBC64E4A8BBF72349637CC07577F714B420B62882
# Client=HOSTNAME@8F81BA58956F8F42@8932D08C49CA76BD843C51EDD1D6640510FA032A7A2403E572BBDA2E5C6B753991CF7E091141D20A2499C5CD3E14C1639D17482E14E1548E5246ACF4E7193D524CDDAC9C9D6A9A36C596B4ECC68BEB0C5BB7082224946FC98E3ADE214EA1343E2DA8DF4229D4D8572AD8679228928A787B6E5390D3A713102FFCC9D0B2188C92
Client=rhc2.sys.local@F1DF72033799940C@37FC42534A812B2351007A24820537466495F97ED352EFC1D9DCAEACBBF5CB98AEF183057CE6D101151F112693C2DAE361435CED1C95E822272FE287A56B4D38EE91B00830A56AE2F26E4738DF099CAEF3372342BE0ACDB78C12FD176EED1FBA376A0399537F848B6FA9AD4E61E6C771A5566F43D62C1F9836AB976CB1111545
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
iD8DBQFNEI7oerU1Wrr7a5ERAkWqAJ9sZEuRLp8rPOjXdUokT03bEfjGuwCfa+Tr
pDK7/KmGj3Hx8vRMufxNx7A=
=zI4S
-----END PGP SIGNATURE-----

The following steps are always required when you've made changes to the configuration files.

Edit /etc/yulerc and remove the first 3 and last 7 lines, this is the GPG/PGP signature.

Example:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
There will be another line here later on.
#####################################################################
#
# Configuration file template for yule.
#
#####################################################################


Lots of Yule configuration removed ...


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
iD8DBQFNEI7oerU1Wrr7a5ERAkWqAJ9sZEuRLp8rPOjXdUokT03bEfjGuwCfa+Tr
pDK7/KmGj3Hx8vRMufxNx7A=
=zI4S
-----END PGP SIGNATURE-----

Then sign the configuration file again with the user yule and copy the file in place as the root user:

su - yule
gpg -o yulerc.asc -a --clearsign --not-dash-escaped /etc/yulerc <-- Type in the passphrase we set earlier.
exit
/bin/mv /home/yule/yulerc.asc /etc/yulerc
service yule reload

A Little More Work On The Client

We will need to create the configuration file and embed it into a postscript file. Make sure you have Imagemagick installed, as you will need convert.

Go and download a good looking picture like http://apod.nasa.gov/apod/image/0903/tycho_chandra_big.jpg. You will want at least a 200K size image, if not larger, to hide the configuration file in it. Also, it is handy to have an original configuration file as backup.

NOTE: The following steps has to be done each time you wish to modify the configuration file of the client.

cd TOPLEVEL_OF_SOURCE_DIR
wget http://apod.nasa.gov/apod/image/0903/tycho_chandra_big.jpg
convert tycho_chandra_big.jpg tycho_chandra_big.ps <-- Convert the JPG to a postscript file.
cp samhainrc.linux rc.`hostname` <-- Get a default configuration.
gpg -a --clearsign --not-dash-escaped rc.`hostname` <-- Clear sign the configuration.
mv rc.`hostname`.asc rc.`hostname` <-- Move the signed file to the normal file name for the configuration file.
/usr/local/sbin/john_stealth -s tycho_chandra_big.ps rc.`hostname`<-- Steganographically hide the configuration file inside the postscript file.
rm rc.`hostname` tycho_chandra_big.* <-- Remove the "clean" files.

Make sure that the resulting postscript file is not very large, or Samhain will fail to download it. I do not have exact numbers, but from experience 66Mb is too large :-)

Now copy the file over to your server:

scp tycho_chandra_big.ps YULE_SERVER:~/rc.`hostname` <-- Assuming scp with root. In real life, please do not open ssh for root.

Back to the server:

cp ~/rc.CliENT_FQDN /var/lib/yule/ <-- Make sure to fill in the clients FQDN.
chown yule:yule /var/lib/yule/rc.CliENT_FQDN <-- Make sure to fill in the clients FQDN.

Back to the client:

/usr/local/sbin/john -t init -p info

This will now build the database in /var/lib/john. Don't worry about all the output at this stage, we are just getting things up and running now.

After we have a database, we have to sign it and copy it over to the server.

gpg -a --clearsign --not-dash-escaped /var/lib/john/john
scp /var/lib/john/john.asc YULE_SERVER:~/file.`hostname` <-- Assuming scp with root. In real life, please do not open ssh for root.
rm /var/lib/john/john*

mv ~/file.CliENT_FQDN /var/lib/yule
chown yule:yule /var/lib/yule/*

It is important that all configuration files start with rc and all database files start with file.

Troubleshooting

Trouble, what trouble?

1. Start with tailing the log file on the server : tail -f /var/log/yule/yule_log
2. Change the log level in /etc/yulerc to info or above (always remember to re-sign the configuration file as described).
3. Recompile without some of the options to test.
4. Have a look at this link : http://www.la-samhna.de/samhain/s_documentation.html

Clean Up

Now we don't want to be leaving breadcrumbs behind us, some clean up is required.

1. Delete all the source files and any tarballs that was downloaded if you built directly on a production server.
2. Delete all entries from your shell history.
3. Remove all the development packages that was installed if you built directly on a production server.
4. Remove /usr/local/sbin/john_stealth and /usr/local/sbin/john_setpwd.

Basically, get rid of any evidence of what you just did.

Tuning

Arguably, this is where the guide should start. Samhain does not understand what is right and what is wrong for this particular server. As such, you need to tune it. The simplest way is to build Samhain without any options what so ever like:

./configure
make
mkdir /var/lib/samhain/

Put the configuration file in /etc/samhainrc, and run

samhain -t init -p info > my_output 2>&1

You can then examine the output file and make the appropriate changes to the Samhain configuration file. The database will be created in /var/lib/samhain. Do not run samhain -t init more than once without deleting the database.

Once you are happy with the configuration, build Samhain in server / client mode.

NOTE: It is however rather important that you profile your server and tune Samhain before it is connected to the Internet.

Honey Pot!

Now for a bit of fun. We really do want intruders to let us know they are on our system. So, we create 2 (or more) files with catchy names and tell Samhain to monitor those files for any changes (that includes access times).

cp /etc/passwd /home/cracked_passwords
cp /etc/hosts /home/customers/credit_cards_2008.xls

Now, in Samhain's configuration file, there is a section called [IgnoreNone], add these files in that section. You can test this by simply catting those files and then run the check. The output should be something like:

CRIT : [2009-04-27T21:33:11+0100] msg=<POliCY [User1] --------T->, path=</home/cracked_passwords>, atime_old=<[2009-04-27T20:25:39]>,
atime_new=[2009-04-27T20:32:37]>,

Nagios Integration

I have not tested this yet, this is just on top of my head, so it may well be very wrong.

So now we have alerts for when things go wrong. By default, the standard Nagios plugin pack ships with check_log. Our Nagios check command will look something like:

check_log -F /var/log/yule_log -O /var/log/yule/yule_nagios_diff_log -q "ERROR|CRIT|ALERT"

You will need to modify how to alert on this particular service. By default Nagios will check 3 times before alerting, but with check_log you will never get an alert. The reason is as follows:

1. check 1: The check returns an error, as it spotted your query (lets say CRIT) in the difference from the old stored log file and the current running log file. The check command now updates the old stored log file.
2. check 2: There is no longer a difference between the old stored log file and the current running one, thus the check passes OK.

Either modify Nagios to alert after a single failure, or write a wrapper specifically for this check to create a lock file somewhere. You then check for this lock file and alert if it exists. Both approaches have some down sides. If we alert on a single check, be prepared for false alerts due to packet loss or a shift in the force. If we create a lock file, you will have to manually remove it.

Now that we are monitoring the log file for changes detected, we also need to monitor that the client process is still up and running. Of course, you will also want to monitor that the server process is running all the time.

I am sure someone will come up with a better way of Nagios integration, like I said, this is just thinking out loud.

What It All Means

At the end of the day, the clear text configuration of each machine being monitored, is neither kept on the client nor on the server. The clear text configuration files should be kept on a different machine inside an encrypted partition.

Nagios makes sure we are alerted of anything (via e-mail or SMS) and hopefully, an intruder will bite on the honey so that we can see him, potentially, even quicker.

Further more, you can not access any help files (such as ./samhain --help or man pages) to indicate that there is a HIDS running on the client.

Of course, if you get access to the server, you can see all the clients who logs in. There are further compiler options so that the logs are also encrypted.

Layers

In the voice of Shrek: "Security is like an onion, it has many layers." Remember that host based intrusion detection is just one more layer in this onion. You also need a good firewall, network intrusion detection, monitoring, centralised logging, log analysis, TCP wrappers, SELinux (or some other mandatory access control mechanism), brute force blockers like fail2ban and much more.

As an example of this, the entire host based intrusion detection is rendered moot if the hacker just kills the process and you are not using monitoring to make sure that the service is running.

Please do not hesitate to contact me with any corrections or improvements or even some constructive criticism. :-)