The Perfect Server - Debian 9 (Nginx, BIND, Dovecot, ISPConfig 3.1)

This tutorial shows how to prepare a Debian 9 server (with Nginx, BIND, Dovecot) for the installation of ISPConfig 3.1, and how to install ISPConfig 3.1. ISPConfig 3 is a web hosting control panel that allows you to configure the following services through a web browser: Apache or nginx web server, Postfix mail server, Courier or Dovecot IMAP/POP3 server, MySQL, BIND or MyDNS nameserver, PureFTPd, SpamAssassin, ClamAV, and many more. This setup covers Nginx web server, BIND as DNS Server, and Dovecot as IMAP / POP3 server.

1 Preliminary Note

In this tutorial, I use the hostname server1.example.com with the IP address 192.168.1.100 and the gateway 192.168.1.1. These settings might differ for you, so you have to replace them where appropriate. Before proceeding further you need to have a minimal installation of Debian 9. This might be a Debian minimal image from your hosting provider or you use the Minimal Debian Server tutorial to setup the base system.

2 Install the SSH Server

If you did not install the OpenSSH server during the system installation, you can do it now:

apt-get -y install ssh openssh-server

From now on you can use an SSH client such as PuTTY and connect from your workstation to your Debian 9 server and follow the remaining steps from this tutorial.

3 Install a shell text editor (Optional)

I'll use nano text editor in this tutorial. Some users prefer the classic vi editor, therefore I will install both editors here. The default vi program has some strange behavior on Debian and Ubuntu; to fix this, we install vim-nox:

apt-get -y install nano vim-nox

(You don't have to do this if you use a different text editor such as joe.)

4 Configure the Hostname

The hostname of your server should be a subdomain like "server1.example.com". Do not use a domain name without subdomain part like "example.com" as hostname as this will cause problems later with your mail setup. First, you should check the hostname in /etc/hosts and change it when necessary. The line should be: "IP Address - space - full hostname incl. domain - space - subdomain part". Edit /etc/hosts. Make it look like this:

nano /etc/hosts

127.0.0.1       localhost.localdomain   localhost
192.168.1.100   server1.example.com     server1

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Then edit the /etc/hostname file:

nano /etc/hostname

It shall contain only the subdomain part, in our case:

server1

Finally, reboot the server to apply the change:

reboot

Log in again and check if the hostname is correct now with these commands:

hostname
hostname -f

The output shall be like this:

root@server1:/tmp# hostname
server1
root@server1:/tmp# hostname -f
server1.example.com

5 Update Your Debian Installation

First make sure that your /etc/apt/sources.list contains the stretch/updates repository (this makes sure you always get the newest updates for the ClamAV virus scanner - this project publishes releases very often, and sometimes old versions stop working), and that the contrib and non-free repositories are enabled.

nano /etc/apt/sources.list

deb http://ftp.us.debian.org/debian/ stretch main contrib non-free
deb-src http://ftp.us.debian.org/debian/ stretch main contrib non-free

deb http://security.debian.org/debian-security stretch/updates main contrib non-free
deb-src http://security.debian.org/debian-security stretch/updates main contrib non-free

IMPORTANT: Add the Debian Backports repository as shown above.

Run

apt-get update

to update the apt package database and

apt-get upgrade

to install the latest updates (if there are any).

6 Change the Default Shell

/bin/sh is a symlink to /bin/dash, however we need /bin/bash, not /bin/dash. Therefore we do this:

dpkg-reconfigure dash

Use dash as the default system shell (/bin/sh)? <-- No

If you don't do this, the ISPConfig installation will fail.

7 Synchronize the System Clock

It is a good idea to synchronize the system clock with an NTP (network time protocol) server over the Internet. Simply run

apt-get install ntp

and your system time will always be in sync.

8 Install Postfix, Dovecot, MySQL, rkhunter and binutils

We can install Postfix, Dovecot, MySQL, rkhunter, and Binutils with a single command:

apt-get install postfix postfix-mysql postfix-doc mariadb-client mariadb-server openssl getmail4 rkhunter binutils dovecot-imapd dovecot-pop3d dovecot-mysql dovecot-sieve dovecot-lmtpd sudo

You will be asked the following questions:

General type of mail configuration: <-- Internet Site
System mail name: <-- server1.example.com

To secure the MariaDB / MySQL installation and to disable the test database, run this command:

mysql_secure_installation

We don't have to change the MySQL root password as we just set a new one during installation. Answer the questions as follows:

Change the root password? [Y/n] <-- y
New password: <-- Enter a new MySQL root password
Re-enter new password: <-- Repeat the MySQL root password
Remove anonymous users? [Y/n] <-- y
Disallow root login remotely? [Y/n] <-- y
Remove test database and access to it? [Y/n] <-- y
Reload privilege tables now? [Y/n] <-- y

Next, open the TLS/SSL and submission ports in Postfix:

nano /etc/postfix/master.cf

Uncomment the submission and smtps sections as follows (leave -o milter_macro_daemon_name=ORIGINATING as we don't need it):

[...]
submission inet n - y - - smtpd
 -o syslog_name=postfix/submission
 -o smtpd_tls_security_level=encrypt
 -o smtpd_sasl_auth_enable=yes
 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
smtps inet n - y - - smtpd
 -o syslog_name=postfix/smtps
 -o smtpd_tls_wrappermode=yes
 -o smtpd_sasl_auth_enable=yes
 -o smtpd_client_restrictions=permit_sasl_authenticated,reject
# -o smtpd_reject_unlisted_recipient=no
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
# -o smtpd_recipient_restrictions=
# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
[...]

Restart Postfix afterwards:

service postfix restart

We want MariaDB to listen on all interfaces, not just localhost, therefore we edit /etc/mysql/mariadb.conf.d/50-server.cnf and comment out the line bind-address = 127.0.0.1  and add the line sql-mode="NO_ENGINE_SUBSTITUTION":

nano /etc/mysql/mariadb.conf.d/50-server.cnf

[...]

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address           = 127.0.0.1

sql-mode="NO_ENGINE_SUBSTITUTION"

[...]

Set the password authentication method in MariaDB to native so we can use PHPMyAdmin later to connect as root user:

echo "update mysql.user set plugin = 'mysql_native_password' where user='root';" | mysql -u root

Edit the file /etc/mysql/debian.cnf and set the MYSQL / MariaDB root password there twice in the rows that start with password.

nano /etc/mysql/debian.cnf

The MySQL root password that needs to be added is shown in red, in this example, the password is "howtoforge".

# Automatically generated for Debian scripts. DO NOT TOUCH!
[client]
host = localhost
user = root
password = howtoforge
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = root
password = howtoforge
socket = /var/run/mysqld/mysqld.sock
basedir = /usr

To prevent the error 'Error in accept: Too many open files' we will set higher open file limits for MariaDB now.

Open the file /etc/security/limits.conf with an editor:

nano /etc/security/limits.conf

and add these lines at the end of the file.

mysql soft nofile 65535
mysql hard nofile 65535

Next, create a new directory /etc/systemd/system/mysql.service.d/ with the mkdir command.

mkdir -p /etc/systemd/system/mysql.service.d/

and add a new file inside:

nano /etc/systemd/system/mysql.service.d/limits.conf

paste the following lines into that file:

[Service]
LimitNOFILE=infinity

Save the file and close the nano editor.

Then we reload systemd and restart MariaDB:

systemctl daemon-reload
service mysql restart

Now check that networking is enabled. Run

netstat -tap | grep mysql

The output should look like this:

root@server1:~# netstat -tap | grep mysql
tcp6       0      0 [::]:mysql              [::]:*                  LISTEN      4027/mysqld
root@server1:~#

9 Install Amavisd-new, SpamAssassin, and ClamAV

To install amavisd-new, SpamAssassin, and ClamAV, we run:

apt-get install amavisd-new spamassassin clamav clamav-daemon zoo unzip bzip2 arj nomarch lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl clamav-docs daemon libio-string-perl libio-socket-ssl-perl libnet-ident-perl zip libnet-dns-perl libdbd-mysql-perl postgrey

The ISPConfig 3 setup uses amavisd which loads the SpamAssassin filter library internally, so we can stop SpamAssassin to free up some RAM:

service spamassassin stop
systemctl disable spamassassin

9.1 Install Metronome XMPP Server (optional)

This step installs the Metronome XMPP Server which provides a chat server that is compatible with the XMPP protocol. This step is optional, if you do not need a chat server, then you can skip this step. No other ISPConfig functions depend on this software.

Add the Prosody package repository in Debian.

echo "deb http://packages.prosody.im/debian stretch main" > /etc/apt/sources.list.d/metronome.list
wget http://prosody.im/files/prosody-debian-packages.key -O - | sudo apt-key add -

Update the package list:

apt-get update

Install the programs that are required for the build process

apt-get install build-essential

and install the packages with apt.

apt-get install git lua5.1 liblua5.1-0-dev lua-filesystem libidn11-dev libssl-dev lua-zlib lua-expat lua-event lua-bitop lua-socket lua-sec luarocks luarocks

luarocks install lpc

Add a shell user for Metronome.

adduser --no-create-home --disabled-login --gecos 'Metronome' metronome

Download Metronome to the /opt directory and compile it.

cd /opt; git clone https://github.com/maranda/metronome.git metronome
cd ./metronome; ./configure --ostype=debian --prefix=/usr
make
make install

Metronome has now be installed to /opt/metronome.