How to Monitor Log Files with Graylog v3.1 on Debian 10

Graylog is a free and open-source log management tool based on Java, ElasticSearch and MongoDB. Graylog can be used to collect, index and analyze any server log from a centralized location or distributed location. We can easily monitor any unusual activity for debugging applications and logs using Graylog. Graylog provides a powerful query language, alerting abilities, a processing pipeline for data transformation and much more. We also can extend the functionality of Graylog through a REST API and Add-ons.

At this moment there is no official guide of Graylog v3.1 on Debian 10 yet.

Installation of Graylog v3.1 on Debian 10 is in 9 steps:

• Step 1 : Update Systems with Debian Backport repositories
• Step 2 : Install some helper
• Step 3 : Install headless JAVA runtime v11.00
• Step 4 : Install MongoDB v4.2, a database to stores the configurations and meta information.
• Step 5 : Install Elasticsearch-OSS 6.x: It stores all the incoming messages and provide a searching facility.
• Step 6 : Install Graylog v3.1 - It receives and log from various inputs and provide a web interface for analysis and monitoring.
• Step 7 : Configure Graylog
• Step 8 : Test Graylog
• Step 9 : Login Graylog

Prerequisite

• A minimal Debian 10. We may refer to this tutorial.
• Minimum 4 GB RAM, 2 core CPU, and 20GB disks
• Default Password : KataLaluan
• Default Secret : SecretRahsiaSecreta
• root access using "su -", Debian recently changes su command behavior. now 'su' command don't replace PATH. use "su -" instead.

Step 1: Update Systems with Debian Backport

Configure system to use the Debian backports repository

cat > /etc/apt/sources.list << EOF
deb http://ftp.debian.org/debian/ buster main contrib non-free
deb http://security.debian.org/debian-security buster/updates main contrib non-free
deb http://ftp.debian.org/debian/ buster-updates main contrib non-free
deb http://ftp.debian.org/debian buster-backports main contrib non-free
EOF
apt -y update
apt -y dist-upgrade

Step 2 - Install headless Java runtime v11.00

Graylog and Elasticsearch is a Java-based application. So, we will need to install Java to your system. By default, the latest version of Java is available in the Debian 10 default repository. We can install it by just running the following command:

apt -y install apt-transport-https default-jdk

Step 3 - Install some helper

We need to install few useful tools as a helper in the process:

• GnuPG - an implementation of the OpenPGP standard, to help in key management system
• wget - a tool to retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols

apt -y install gnupg wget

Step 4 - Install MongoDB v4.2

By default, MongoDB is not available in the Debian 10 default repository. So we need to add MongoDB repository to the system:

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 4B7C549A058F8B6B
echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.2 main" | tee /etc/apt/sources.list.d/latest-mongodb.list
apt -y update
apt install -y mongodb-org

Enable and restart MongoDB services:

systemctl enable mongod.service
systemctl start mongod.service

Step 5 : Install Elasticsearch-OSS 6.x

At this moment, Graylog v3.1 is not support Elasticsearch-OSS 7.x yet

We're going to add the Elasticsearch key and repository to the Debian. With the elasticsearch repository provided by the elastic.co, we're able to install the Elasticsearch by running the following command:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | tee -a /etc/apt/sources.list.d/latest-elastic-6.x.list
apt -y update
apt -y install elasticsearch-oss

Configure Elasticsearch for the cluster name

sed -i "s/#cluster.name: my-application/cluster.name: graylog-application/g" /etc/elasticsearch/elasticsearch.yml

Enable and restart Elasticsearch services:

systemctl enable elasticsearch.service
systemctl start elasticsearch.service

Step 6 : Install Graylog v3.1

We going to download a simple Graylog package that help to add Graylog key and configure the Graylog repository

cd /tmp/
wget https://packages.graylog2.org/repo/packages/graylog-3.1-repository_latest.deb
dpkg -i graylog-3.1-repository_latest.deb
apt -y update

Install Graylog by running the following command:

apt -y install graylog-server

Step 7 : Configure Graylog

Hash the password, and copy the hash. "KataLaluan" is the current selected password.

echo 'KataLaluan' | tr -d '\n' | sha256sum | cut -d" " -f1

Add the hashed password in the Graylog configuration file

sed -i "s/^root_password_sha2 =\$/root_password_sha2 = a25d2f6605c9e27f182d39b66a8b527eb7f2360e52b2ccc7614f8ac24e472bef/g" /etc/graylog/server/server.conf

Add the secret in the Graylog configuration file, The minimum length of it is 16 characters.

sed -i "s/^password_secret =\$/password_secret = SecretRahsiaSecreta/g" /etc/graylog/server/server.conf

Allow external access to graylog

sed -i "s/^#http_bind_address = 127.0.0.1:9000/http_bind_address = 0.0.0.0:9000/g" /etc/graylog/server/server.conf

Change the timezone accroding to location

sed -i "s/#root_timezone = UTC/root_timezone = Asia\/Kuala_Lumpur/g" /etc/graylog/server/server.conf

Enable and restart Graylog services:

systemctl enable graylog-server.service
systemctl start graylog-server.service

if the Graylog is behind a router, we must set the router WAN's IP address into the Graylog configuration. It also can be a DNS's A records that point to the same IP address

sed -i '/http_publish_uri =/c\http_publish_uri = http://graylog.howtoforge.com:9000/' /etc/graylog/server/server.conf

Step 8 : Test Graylog

Lets test the Graylog using some primitive commands

apt -y install netcat curl

Here is some sample command to log.

echo "Hello Graylog, let's be friends." | nc -w 1 -u 127.0.0.1 9099

Here is some sample command to get Graylog server API status.

curl -X GET http://localhost:9200
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Here is some sample command to get Graylog server log.

tail -f /var/log/graylog-server/server.log

Step 9 : Login Graylog

Let used the WebGUI. the URL can be:

• http://<Local_IP_Address>:9000/
• http://<Public_IP_Address>:9000/
• http://<A_Record>:9000/

Sample of the URL

• http://192.168.0.3:9000/
• http://104.26.2.165:9000/
• http://graylog.howtoforge:9000/

After enter the URL in a browser, we should see the following sign-in page, the default username is admin, and the selected password is KataLaluan,

After login, we should see the following Graylog page:

Conclusion

Done, we have successfully installed and configured Graylog 3.1 server on Debian 10. Now we can easily see the logs and analysis of the system logs at the central location. Get more information from the Graylog documentation page. Please comment and feedback if have any question.

Happy logging.