How to Install Malcolm Network Traffic Analysis Tool on Ubuntu 22.04

Malcolm is a simple, easy-to-use, and powerful network traffic analysis tool. It is capable of capturing artifacts (PCAP files) and Zeek logs. Malcolm provides an interconnected framework that makes it greater than the sum of its parts. Malcolm's easy, flexible deployment and robust combination of tools fill a void in the network security space and make advanced network traffic analysis accessible to many in both the public and private sectors.

In this post, we will show you how to install the Malcolm network traffic analysis tool on Ubuntu 22.04.

Prerequisites

  • A server running Ubuntu 22.04.
  • Minimum 16 GB RAM and 4 CPU core.
  • A root password is configured on the server.

Create a Malcolm System User

First, you will need to create a dedicated user account to run Malcolm. You can create it with the following command:

useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolm

Next, set the user password with the following command:

passwd malcolm

Next, check the UID of the Malcolm user with the following command:

id malcolm

You should see the following output:

uid=1000(malcolm) gid=1000(malcolm) groups=1000(malcolm),27(sudo)

Install Malcolm Ubuntu 22.04

First, switch the user to Malcolm and download the latest version of Malcolm using the following command:

su - malcolm
git clone https://github.com/idaholab/Malcolm

Once the download is completed, change the directory to the downloaded directory and start installing Malcolm using the following command:

cd Malcolm
sudo ./scripts/install.py

During the installation, you will be asked several questions, as shown below:

Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog']

"docker info" failed, attempt to install Docker? (Y/n): Y

Attempt to install Docker using official repositories? (Y/n): Y
Installing required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common']


Installing docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io']
Installation of docker packages apparently succeeded

Add a non-root user to the "docker" group?: Y

Enter user account: malcolm

Add another non-root user to the "docker" group?: n

"docker-compose version" failed, attempt to install docker-compose? (Y/n): Y

Install docker-compose directly from docker github? (Y/n): Y
Download and installation of docker-compose apparently succeeded


fs.file-max increases allowed maximum for file handles
fs.file-max= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y


fs.inotify.max_user_watches increases allowed maximum for monitored files
fs.inotify.max_user_watches= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y


fs.inotify.max_queued_events increases queue size for monitored files
fs.inotify.max_queued_events= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y


fs.inotify.max_user_instances increases allowed maximum monitor file watchers
fs.inotify.max_user_instances= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.max_map_count increases allowed maximum for memory segments
vm.max_map_count= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y


net.core.somaxconn increases allowed maximum for socket connections
net.core.somaxconn= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.dirty_background_ratio defines the percentage of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

vm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything
vm.dirty_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y

/etc/security/limits.d/limits.conf increases the allowed maximums for file handles and memlocked segments
/etc/security/limits.d/limits.conf does not exist, create it? (Y/n): Y

Once the Malcolm is installed, you can proceed to the next step.

Configure Malcolm

After installing Malcolm, you will need to configure it using the following command:

sudo ./scripts/install.py --configure

During the Malcolm configuration, you will be asked for several questions. Answer all questions as shown below:

  • Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): Y
  • Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yes
  • Setting 3 workers for Logstash pipelines. Is this OK? (Y/n): yes
  • Restart Malcolm upon system or Docker daemon restart? Yes Then, Choose the default option, unless-stopped.
  • Choose whether to setup Malcolm with HTTPS: Yes
  • Choose whether Malcolm will run behind any proxy: No
  • Choose networking: Just enter to accept the defaults. Choose LDAP: No
  • Store OpenSearch index snapshosts locally in /opt/malcolm/Malcom/opensearch-backup? Yes
  • Choose to Compress OpenSearch index snapshots: Yes
  • Whether you want to delete the oldest indices when the database exceeds a certain size: No
  • Reverse DNS lookup locally for source and destination IP addresses in logs: No
  • Hardware vendor OUI lookups for MAC addresses: Yes
  • Perform string randomness scoring on some fields: yes
  • Expose OpenSearch port to external hosts: no
  • Expose Logstash port to external hosts: no
  • Forward Logstash logs to external OpenSearch instance: no
  • Expose Filebeat TCP port to external hosts: no
  • Enable file extraction with Zeek: yes
  • Choose interesting as the extraction behavior. Choose file preservation method: quarantine
  • Scan extracted files/PE files with ClamAV: yes
  • Scan extracted files/PE files with Yara: yes
  • Scan extracted files/PE files with Capa: yes
  • Lookup extracted file hashes with VirusTotal: no
  • Download updated scanner signatures periodically: yes
  • Should Malcolm capture network traffic to PCAP files: yes
  • Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: eth0
  • Capture packets using netsniff-ng? (Y/n): yes
  • Capture packets using tcpdump? (y/N): no
  • Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) (): You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005): not port 9200 and not port 5044 and not port 8005
  • Disable capture interface hardware offloading and adjust ring buffer sizes? (y/N): n

Once the Malcolm has been configured, restart your system to apply the configuration.

sudo reboot

Create Malcolm Admin Account

Next, you will need to create an administrative account to access Malcolm web interface.

First, switch the user to Malcolm and navigate to the Malcolm directory using the following command:

su - malcolm
cd ~/Malcolm

Next, run the following command to create an admin account:

./scripts/auth_setup

Answer all questions as shown below to create an admin account:

  • Store administrator username/password for local Malcolm access? yes
  • Set administrator password and username.
  • (Re)generate self-signed SSL certs for web traffic HTTPS: yes
  • (Re)generate self-signed certificates for a remote log forwarder: yes
  • Store username/password for forwarding Logstash events to a secondary, external OpenSearch instance: no. Store username/password for email alert sender account: no

Pull Malcolm Docker Images

Next, you will need to download all required Docker images from the Docker Hub registry. You can download all with the following command:

docker-compose pull

Once all the images are downloaded, verify all the images using the following command:

docker images

You should get the following output:

REPOSITORY                        TAG       IMAGE ID       CREATED       SIZE
malcolmnetsec/filebeat-oss        6.2.0     5e9fa4c8ea2d   11 days ago   648MB
malcolmnetsec/arkime              6.2.0     4f4e6025c82d   11 days ago   793MB
malcolmnetsec/zeek                6.2.0     5b117ad2b4bb   11 days ago   1.4GB
malcolmnetsec/dashboards          6.2.0     9dcaff859eec   11 days ago   1.13GB
malcolmnetsec/logstash-oss        6.2.0     cf4d75dcf4af   11 days ago   1.64GB
malcolmnetsec/file-monitor        6.2.0     26227c3c7dc9   11 days ago   589MB
malcolmnetsec/nginx-proxy         6.2.0     3b8b5413b52e   11 days ago   121MB
malcolmnetsec/file-upload         6.2.0     2c704be24433   11 days ago   263MB
malcolmnetsec/api                 6.2.0     90f964b5d728   11 days ago   148MB
malcolmnetsec/suricata            6.2.0     cbdb091d2df4   11 days ago   274MB
malcolmnetsec/htadmin             6.2.0     2b55630700d1   11 days ago   242MB
malcolmnetsec/opensearch          6.2.0     9ccb4665bd6c   11 days ago   1.25GB
malcolmnetsec/pcap-monitor        6.2.0     d957d803bdbb   11 days ago   214MB
malcolmnetsec/freq                6.2.0     3959a9daa952   11 days ago   131MB
malcolmnetsec/dashboards-helper   6.2.0     7da9699a72b3   11 days ago   168MB
malcolmnetsec/pcap-capture        6.2.0     df63e8daa369   11 days ago   121MB
malcolmnetsec/name-map-ui         6.2.0     6bd6c7c58d36   11 days ago   120MB

Start Malcolm Service

At this point, all required components for Malcolm are ready. You can now start the Malcolm service using the following command:

./scripts/start

Wait for some time to start all services. Once all the services are started, verify all the running services using the following command:

docker ps -a

You should see all running containers in the following output:

CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS                             PORTS                                                                                            NAMES
840ea2b0e9ad   malcolmnetsec/nginx-proxy:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          0.0.0.0:443->443/tcp, 127.0.0.1:5601->5601/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:9200->9200/tcp   malcolm-nginx-proxy-1
dd5c8c63816c   malcolmnetsec/suricata:6.2.0            "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)                                                                                                           malcolm-suricata-1
3112e1bd8f73   malcolmnetsec/filebeat-oss:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          127.0.0.1:5045->5045/tcp                                                                         malcolm-filebeat-1
c93cfe93ad7e   malcolmnetsec/file-upload:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          80/tcp, 127.0.0.1:8022->22/tcp                                                                   malcolm-upload-1
18ee20b46f3c   malcolmnetsec/dashboards:6.2.0          "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          5601/tcp                                                                                         malcolm-dashboards-1
2c34206c06e4   malcolmnetsec/zeek:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)                                                                                                           malcolm-zeek-1
41103ef99ce1   malcolmnetsec/logstash-oss:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp                                                     malcolm-logstash-1
0408f42a76c3   malcolmnetsec/dashboards-helper:6.2.0   "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          28991/tcp                                                                                        malcolm-dashboards-helper-1
3e78024620de   malcolmnetsec/arkime:6.2.0              "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          8000/tcp, 8005/tcp, 8081/tcp                                                                     malcolm-arkime-1
58cd869beced   malcolmnetsec/pcap-monitor:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          30441/tcp                                                                                        malcolm-pcap-monitor-1
1040fa8bd6df   malcolmnetsec/file-monitor:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          3310/tcp, 8440/tcp                                                                               malcolm-file-monitor-1
25c83f14413d   malcolmnetsec/zeek:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-zeek-live-1
b321a96c0362   malcolmnetsec/api:6.2.0                 "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          5000/tcp                                                                                         malcolm-api-1
0f1f4ac023f9   malcolmnetsec/name-map-ui:6.2.0         "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          8080/tcp                                                                                         malcolm-name-map-ui-1
ba4d553cf6b5   malcolmnetsec/suricata:6.2.0            "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-suricata-live-1
e4637d0ec04d   malcolmnetsec/opensearch:6.2.0          "/usr/local/bin/dock…"   17 minutes ago   Up 13 minutes (health: starting)   9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp                                                           malcolm-opensearch-1
ac002e31d9be   malcolmnetsec/htadmin:6.2.0             "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          80/tcp                                                                                           malcolm-htadmin-1
7223d5244a7b   malcolmnetsec/pcap-capture:6.2.0        "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes                                                                                                                       malcolm-pcap-capture-1
971931b21788   malcolmnetsec/freq:6.2.0                "/usr/local/bin/dock…"   17 minutes ago   Up 17 minutes (unhealthy)          10004/tcp                                                                                        malcolm-freq-1

You can also verify all listening ports using the following command:

ss -atlnp | grep -i docker

You should see the following output:

LISTEN 0      65535      127.0.0.1:5601       0.0.0.0:*    users:(("docker-proxy",pid=7480,fd=4))   
LISTEN 0      65535        0.0.0.0:488        0.0.0.0:*    users:(("docker-proxy",pid=7519,fd=4))   
LISTEN 0      65535      127.0.0.1:9200       0.0.0.0:*    users:(("docker-proxy",pid=7443,fd=4))   
LISTEN 0      65535      127.0.0.1:5044       0.0.0.0:*    users:(("docker-proxy",pid=6247,fd=4))   
LISTEN 0      65535      127.0.0.1:5045       0.0.0.0:*    users:(("docker-proxy",pid=7063,fd=4))   
LISTEN 0      65535      127.0.0.1:8022       0.0.0.0:*    users:(("docker-proxy",pid=6826,fd=4))   
LISTEN 0      65535        0.0.0.0:443        0.0.0.0:*    users:(("docker-proxy",pid=7567,fd=4))

Access Malcolm

You can now access the Malcolm OpenSearch dashboard using the URL https://your-server-ip/dashboards/. You will be asked to provide your admin username and password:

Provide your admin username, password and click on the Sign in button. You should see the OpenSearch dashboard on the following screen:

To access the Malcolm Capture File and Log Archive Upload screen, type the URL https://your-server-ip/upload/.

To access the Host and subnet name mapping editor, type the URL https://your-server-ip/name-map-ui/.

To access the Account Management screen, use the URL https://your-server-ip:488/

Congratulations! you have successfully installed and configured Malcolm network traffic analysis tool on Ubuntu 22.04.

Share this page:

Suggested articles

0 Comment(s)

Add comment