How to Install Logstash on Ubuntu 18.04

Logstash is a free and open-source tool, and world’s most popular log analysis platform for collecting, parsing, and storing logs for future use. Logstash comes with a rich set of plugins and a very expressive template language that makes it easy to transform data streams. Logstash is a part of the ELK stack, but you can also use it independently. Logstash has an ability to pull from any data source using input plugins, apply a wide variety of data transformations and ship the data to a large number of destinations using output plugins.

In this tutorial, we will show you how to install and configure Logstash on Ubuntu 18.04 server.

Prerequisites

• A server running Ubuntu 18.04.
• A root password is configured on your server.

Getting Started

Before starting, update your system with the latest version. You can do it by running the following command:

apt-get update -y
apt-get upgrade -y

Once your system is updated, restart it to apply the changes.

Install Java

Before installing Java, make sure you have either Java 8 or Java 11 installed. You can install Java 8 with the following command:

apt-get install openjdk-8-jdk -y

After installing Java, check the version of Java using the following command:

java -version

You should see the following output:

openjdk version "1.8.0_162"
OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-1-b12)
OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode)

Install Logstash

Before installing Logstash, you will need to install Elasticsearch on your server. By default, Elasticsearch is not available in the Ubuntu 18.04 server. So you will need to add the repository for Elasticsearch.

First, install the required package with the following command:

apt-get install apt-transport-https -y

Next, download and add the Elasticsearch GPG key with the following command:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -

Next, add the repository with the following command:

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

Next, install Elasticsearch and Logstash with the following command:

apt-get install elasticsearch logstash -y

Once both packages are installed, start Logstash and Elasticsearch, and enable them to start after system reboot with the following command:

systemctl start logstash
systemctl enable logstash
systemctl start elasticsearch
systemctl enable elasticsearch

You can also check the status of Logstash with the following command:

systemctl status logstash

You should see the following output:

? logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-11-20 10:13:22 UTC; 5s ago
 Main PID: 3790 (java)
    Tasks: 13 (limit: 1110)
   CGroup: /system.slice/logstash.service
           ??3790 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 

Nov 20 10:13:22 ubuntu systemd[1]: Started logstash.

By default, Elasticsearch listen on port 9200. You can check it with the following command:

netstat -ant | grep 9200

You should see the following output:

tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN     
tcp6       0      0 ::1:9200                :::*                    LISTEN  

Configure Logstash

Logstash configuration is one of the most challenging parts of any system administrator. Logstash configuration consists of three main configuration sections, Logstash Inputs, Logstash Filters and Logstash Outputs. Each section responsible for different functions and using different Logstash plugins.

Configure Logstash Input

First, you will need to create an input configuration file beats-input.conf to receive data from Beats on TCP port 5044:

nano /etc/logstash/conf.d/beats-input.conf

Add the following lines:

input {
  beats {
    port => 5044
  }
}

Save and close the file when you are finished.

Configure Logstash Filters

Next, you will need to configure a filter plugin to process events received from the beats. Here, we will configure beats to collect SSH authentication events from Ubuntu systems. To do so, create a new filter configuration file ssh-auth-filter.conf:

nano /etc/logstash/conf.d/ssh-auth-filter.conf

Add the following lines:

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:dst_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:src_host}.*" }
    add_field => { "activity" => "SSH Logins" }
    add_tag => "linux_auth"
    }
}

Save and close the file when you are finished.

Configure Logstash Output

Next, you will need to configure output plugins to enables Logstash to sent event data to specific destinations.

Here, we will create a Logstash output configuration file that sents data to Elasticsearch running on a localhost.

nano /etc/logstash/conf.d/elasticsearch-output.conf

Add the following lines:

output {
   elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "ssh_auth-%{+YYYY.MM}"
 }
  stdout { codec => rubydebug }
}

Save and close the file when you are finished. Then, restart Logstash service to apply the changes:

systemctl restart logstash

Test Logstash

Logstash is now installed and configured. It's time to test whether it is working or not.

You can verify the Logstash configuration with the following command:

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

If everything is fine, you should see the following output:

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-11-26T06:27:48,104][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2019-11-26T06:27:48,119][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2019-11-26T06:27:50,331][INFO ][org.reflections.Reflections] Reflections took 47 ms to scan 1 urls, producing 20 keys and 40 values 
Configuration OK
[2019-11-26T06:27:51,899][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Congratulations! you have successfully installed and configured Logstash on Ubuntu 18.04 server.