Securing ISPConfig 3 Control Panel (Port 8080) With Let's Encrypt Free SSL

Discussion in 'Tips/Tricks/Mods' started by ahrasis, Feb 14, 2017.

  1. tripp

    tripp New Member

    My setup does the same as zicguy and I agree, excellent work.
    I've also tried everything he has done. As well as going back through all the steps. Nothing seems to work.
    I got tired of messing with it and just re-install Ubuntu with a back-up and everything works now. But for how long and is this going to happen again..?
  2. zicguy

    zicguy Member

    So, it seems i'm not the only one dealing with this problem... In my case, i have this issue on a production server, so I can't easily reinstall my server (FYI, I'm on Debian Wheezy). This issue really makes me crazy. I tried a lot of things as change pemfile to monitrc, put none content on it, etc... Still an old certificate in the browser... I searching with this cert on the filesystem without any success. I really don't understand.
    It seems there is no issue with Monit because Pure-FTP has the same behavior and returns the same wrong cert. So, this issue seems to happen only on services using pemfile. Is there a possibility that Let's encrypt returns a wrong old certificates? Is certificate revocation could solve that? What if I try to revoke, delete and re create cert files on my main domain? If so, what is the right process to achieve that?

    Thanks for you help ;-)
  3. ahrasis

    ahrasis Well-Known Member

    Remember that ispserver.pem is a created file and not the same as the other ispserver.key or ispserver.crt (symlinked) files. If you did not recreate ispserver.pem (and chmod it to 600) after LE SSL certs for the hostname -f fqdn is renewed, then I think it will still use the old file. This is the earlier step, so do check that out too.

    Basically simply run "ls -la /usr/local/ispconfig/interface/ssl/ispserver.pem" will show you what is the date of the file and whether it is just an empty file.

    Reinstalling is never necessary but if after doing so it works, it only shows that you failed to check properly for the original failure.

    By the way, did you restart the services (monit and pure-ftpd-mysql)?
    Last edited: Jul 1, 2018
  4. zicguy

    zicguy Member

    Of course I did. I restart both services many times...
    A new strange thing: I tried to change monit configuration to put a self-signed generated cert:
    set httpd port 2812 and
         SSL ENABLE
         PEMFILE /etc/monit/monit.pem
         allow monitAdmin:XXXXXXXX
    Then, I restart monit service by "service monit restart" command line. Restart was OK, but in browsers, still the old let's encrypt used... It is so strange...
  5. zicguy

    zicguy Member

    Finally, after hours and hours, I found the issue, and the solution :)
    For an unknow reason, I had multiple stucked processes for monit and pure-ftp even if services was stopped. One process in particular, for each of them, was launched few month ago, and I think these processes had the old SSL cert. So, first I stopped both, then I ran
    ps aux | grep monit
    ps aux | grep pure-ftpd-mysql
    and I kill all relevant PID.
    After that, I restarted both services and everything works well now! Monit and Pure-FTP hav the right SSL LE certificate.
    For some reason, and in particular cases, restart services is not sufficient and we also had to check processes and kill them if necessary. I don't know if this could be added in the script. @ahrasis, if you have an idea to enhance the script in this way ;)
    I suspect Monit to manage not very well processes when it restart them... but this is another story... Hope my post could help!
    ahrasis likes this.
  6. ISPConfig Developer ISPConfig Developer

    so what about mysql?
    This works for mariadb 10.3 using debian 9
    assuming the ispcerts have been symlinked according to your script.

    cp /usr/local/ispconfig/interface/ssl/ispserver.pem /etc/mysql/mysql.pem
    openssl rsa -in /usr/local/ispconfig/interface/ssl/ispserver.key -out /etc/mysql/mysql.crt
    chown mysql:mysql /etc/mysql/mysql.crt
    chown mysql:mysql /etc/mysql/mysql.pem
    chmod 400 /etc/mysql/mysql.crt
    chmod 400 /etc/mysql/mysql.pem
    and no, you can't simply symlink those, it won't read.

    change your /etc/mysql/mariadb.conf.d/50-server.cnf
  7. ahrasis

    ahrasis Well-Known Member

    Milllion thanks and credits to for his awesome tip above.

    [Using Let's Encrypt In ISPConfig for MySQL Server]

    7B. I'll suggest a less complicated approach to extend the use of ISPConfig LE ssl certs to mysql server.

    Step A - Enabling SSL With Cipher
    In /etc/mysql/mariadb.conf.d/50-server.cnf (under the [mysqld] section), simply add these following lines:
    You can use this online command to insert the above code:
    sed -i '/\[mysqld\]/a ssl-cert=/etc/mysql/ssl/server-cert.pem\nssl-key=/etc/mysql/ssl/server-key.pem\nssl-cipher=TLSv1.2,TLSv1.3' /etc/mysql/mariadb.conf.d/50-server.cnf
    Step B - Copying SSL Files
    The LE ssl certs are usages are explain in here. Use scp to copy these files from your Server LE ssl folder to mysql folder:
    mkdir -p /etc/mysql/ssl
    cd /usr/local/ispconfig/interface/ssl
    scp ispserver.crt /etc/mysql/ssl/server-cert.pem
    scp ispserver.key /etc/mysql/ssl/server-key.pem
    chown mysql:mysql /etc/mysql/ssl/server*.pem
    chmod 400 /etc/mysql/ssl/server*.pem
    Step C - Restart MySQL Service & Check
    The ssl for mysql should be automatically enabled with the above ssl settings after restarting mysql and check your database for whether the above config is working as follows:
    service mysql restart
    mysql --ssl -uroot -p
    In your mysql, you can enter these command:
    The first one should display something like:
    | Variable_name       | Value                                 |
    | have_openssl        | YES                                   |
    | have_ssl            | YES                                   |
    | ssl_ca              |                                       |
    | ssl_capath          |                                       |
    | ssl_cert            | /etc/mysql/ssl/server-cert.pem        |
    | ssl_cipher          | TLSv1.2,TLSv1.3                       |
    | ssl_crl             |                                       |
    | ssl_crlpath         |                                       |
    | ssl_key             | /etc/mysql/ssl/server-key.pem         |
    | version_ssl_library | OpenSSL 1.1.1-pre8 (beta) 20 Jun 2018 |
    While the second should display something like:
    SSL:                    Cipher in use is TLS_AES_256_GCM_SHA384
    Step D - Copying SSL Files Between Servers
    The same scp command can also be used to transfer files between servers provided you already created the required access keys as suggested in post #203 for multiserver setup by adding the mysql server name or its ip number. If it has the required access keys, you can use the following command follows to remotely create the ssl folder, scp the files and restart mysql:
    ssh [email protected] 'mkdir /etc/mysql/ssl'
    cd /usr/local/ispconfig/interface/ssl
    scp ispserver.crt [email protected]:/etc/mysql/ssl/server-cert.pem
    scp ispserver.key [email protected]:/etc/mysql/ssl/server-key.pem
    ssh [email protected] 'service mysql restart'
    Step E - Automate Future Copying
    To automate this scp in the future (when the LE ssl certs are renewed), you should add the scp lines in /etc/init.d/ like this:
    # Required-Start:  $local_fs $network
    # Required-Stop:  $local_fs
    # Default-Start:  2 3 4 5
    # Default-Stop:  0 1 6
    # Short-Description:  LE ISPSERVER.PEM AUTO UPDATER
    # Description:  Update ispserver.pem automatically after ISPC LE SSL certs are renewed.
    cd /usr/local/ispconfig/interface/ssl/
    mv ispserver.pem ispserver.pem-$(date +"%y%m%d%H%M%S").bak
    cat ispserver.{key,crt} > ispserver.pem
    chmod 600 ispserver.pem
    chmod 600 /etc/ssl/private/pure-ftpd.pem
    scp ispserver.crt /etc/mysql/ssl/server-cert.pem
    scp ispserver.key /etc/mysql/ssl/server-key.pem
    service mysql restart
    service pure-ftpd-mysql restart
    service monit restart
    service postfix restart
    service dovecot restart
    service nginx restart
    Step F - Future Copying Between Servers
    Add server name or its ip address to the scp lines and service mysql restart if you are in the multi server setup, like in the sample codes in Step D above.

    1. I already referred this post in the OT as part of the extended guide for this thread.

    2. This has been added in LE4ISPC script so mysql service will now be covered, secured and updated together with other services.

    3. You may need to create CA and client's key and cert files and add them (to this database server and client server)to enable mysql ssl remote access from another server. Do read good articles about it here and here.

    4. The best way for securing mysql server may be best served by using openssl created ssl certs rather than LE ssl certs.
    Last edited: Sep 20, 2020 likes this.
  8. ISPConfig Developer ISPConfig Developer

    good point using default filenames :)
    however what you suggested with scp didn't work for me, why I had to use the ssl in out command
    could be my mistake ... dunno
  9. ISPConfig Developer ISPConfig Developer

  10. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I have this script to monitor what ssl cert is in use by mysql and rebuild/restart it if it differs from the latest letsencrypt certificate, which might be useful, and I think also implies I had to use openssl to generate, not a simple cp. Save as /usr/local/sbin/
    #  compares the ssl certficate/key used by mysql
    # with the current certificate/key issued by letsencrypt and copy the latter
    # to the former if they differ.
    # this can be run as a cronjob to propogate letsencrypt certificate changes
    # to mysql
    LE_DIR=/etc/letsencrypt/live/`hostname -f`
    OPENSSL=`which openssl 2>/dev/null | head -1`
    # Check if letsencrypt has been setup
    if [ ! -f ${LE_CA} -o ! -f ${LE_CERT} -o ! -f ${LE_KEY} ]
            echo "Letsencrypt files not found.  You must setup letsencrypt and issue a certificate first." 1>&2
            exit 0
    # Check openssl binary exists
    if [ ! -f ${OPENSSL} ]
            echo "Cannot find openssl. Exiting." 1>&2
            exit 1
    # setup_certs() copies/formats the letsencrypt files for mysql
    # (workarounds for file ownership/permissions and built-in YaSSL library limitations)
    function setup_certs() {
            if [ ! -d ${MYSQL_SSLDIR} ]
                    mkdir ${MYSQL_SSLDIR}
                    chown 750 ${MYSQL_SSLDIR}
            # format private key as PKCS1
            ${OPENSSL} rsa -in ${LE_KEY} -out ${MYSQL_KEY} 2>&1 | grep -v 'writing RSA key' 1>&2
            # copy the certificate
            ${OPENSSL} x509 -in ${LE_CERT} -out ${MYSQL_CERT}
            # YaSSL won't follow a certificate chain, it only verifies the certificate
            # against the signing certificate, which is the intermediate certificate
            # letsencrypt used.  If mysqld is linked with openssl, ${SYS_CAFILE}
            # should be used instead, to handle changes of the intermediate.
            cat ${LE_CA} ${SYS_CAFILE} > ${MYSQL_CA}
            chgrp -R mysql ${MYSQL_SSLDIR}
            chmod -R o-rwx ${MYSQL_SSLDIR}
    # restart mysqld if it is running
    function restart_mysqld_if_running() {
            /etc/init.d/mysql status 2>/dev/null >/dev/null
            if [ $? -eq 0 ]
                    /etc/init.d/mysql restart >/dev/null
    if [ ! -f ${MYSQL_CA} -o ! -f ${MYSQL_CERT} -o ! -f ${MYSQL_KEY} ]
            setup_certs && restart_mysqld_if_running
    else # check if keys/certificates changed
            le_modulus=`${OPENSSL} rsa -noout -modulus -in ${LE_KEY} | md5sum`
            my_modulus=`${OPENSSL} rsa -noout -modulus -in ${MYSQL_KEY} | md5sum`
            le_serial=`${OPENSSL} x509 -noout -serial -in ${LE_CERT}`
            my_serial=`${OPENSSL} x509 -noout -serial -in ${MYSQL_CERT}`
            le_ca_serial=`${OPENSSL} x509 -noout -serial -in ${LE_CA}`
            my_ca_serial=`${OPENSSL} x509 -noout -serial -in ${MYSQL_CA}`
            if [ "${le_modulus}" != "${my_modulus}" -o "${le_serial}" != "${my_serial}" -o "${le_ca_serial}" != "${my_ca_serial}" ]
                    setup_certs && restart_mysqld_if_running
    exit 0
    Then just run that from a cronjob, eg. /etc/cron.d/letsencrypt-restarts:
    10 3 * * *  root  /usr/local/sbin/
    ahrasis likes this.
  11. ISPConfig Developer ISPConfig Developer

    using this threads plugin it could also be done using
    echo "/etc/letsencrypt/archive/$(hostname -f)/ IN_MODIFY /usr/local/sbin/" >> /var/spool/incron/root
    I guess :)
    ahrasis likes this.
  12. ahrasis

    ahrasis Well-Known Member

    +1. That is if you prefer using incron like me.

    You can also add the same in the created /etc/init.d/ as the post #247 above now reflects from this thread opening post. I also adjusted it to use /etc/mysql/my.cnf file instead of mariadb since it will cover both mysql and mariadb safely.

    To add and enable the certs settings and links in that file, I added this one line command in the LE4ISPC script for my.cnf file:
    sed -i '/\[mysqld\]/a ssl-cert=/etc/mysql/ssl/server-cert.pem\nssl-key=/etc/mysql/ssl/server-key.pem\nssl-cipher=TLSv1.2,TLSv1.3' /etc/mysql/my.cnf
    For mariadb 50-server.cnf, you can use this one line command instead:
    sed -i '/\[mysqld\]/a ssl-cert=/etc/mysql/ssl/server-cert.pem\nssl-key=/etc/mysql/ssl/server-key.pem\nssl-cipher=TLSv1.2,TLSv1.3' /etc/mysql/mariadb.conf.d/50-server.cnf
    About copying from ISPConfig SSL folder, I tested scp (not cp) again in another server of mine (Ubuntu 18.04), it works just fine and the original files are rightly copied. The reason cp command does not work for this is because it only copied the symlink files instead of the needed original files.
    Last edited: Aug 24, 2018 likes this.
  13. ISPConfig Developer ISPConfig Developer

    well yeah, somehow I need to do the conversion, have you tested with mysql or mariadb? I used maria 10.3
  14. ahrasis

    ahrasis Well-Known Member

    I am using Mariadb 10.3 and the test was done on it. Theoretically it should work on Mysql as well since it is using the same file too.
  15. ISPConfig Developer ISPConfig Developer

    hmm I need to doublecheck my files or what I did I guess ^^
  16. ISPConfig Developer ISPConfig Developer

    well there is an issue for me and incron...

    add a file to watch and define a script to run, restart incron.
    touch / modify the watched file, incron does execute the script like expected.
    now touch / modify the file again and... nothing happens -.- need to add an service restart of incron after finish touching the file :(
  17. ahrasis

    ahrasis Well-Known Member

    Not sure about that. Last time I tested, mine works as many time as the changes are made.

    What did you have / add in the /var/spool/incron/root? Did you give proper permission to root?
  18. ISPConfig Developer ISPConfig Developer

  19. ahrasis

    ahrasis Well-Known Member

    Checking my incron, I use IN_ATTRIB to watch for specific file changes and IN_MODIFY for directory content (file) changes (ends with a slash "/"). Since you are using it for monitoring file changes, I'd suggest to use IN_ATTRIB instead. Examples:
    /etc/nginx/sites-available/domain.tld.vhost IN_ATTRIB /bin/bash /etc/init.d/
    /etc/letsencrypt/archive/server1.domain.tld/ IN_MODIFY /bin/bash /etc/init.d/
    There are some instances in the FAQ where it explains why IN_MODIFY doesn't work. Check this out:
    Do note that while incron is monitoring specified file / directory changes, I also monitor incron itself via Monit where if it never finished causing the use of too much resources etc, Monit will kill it and restart. It did happen to me once.
    Last edited: Jul 20, 2018 likes this.
  20. ISPConfig Developer ISPConfig Developer

    oh crap not only thank you very much for your time explaining incron but also for giving the hints ... I do use monit and hell yes it can check filechanges ... and it is embedded in ispconfig and one could add a button "fixconfigs" ... man good point

Share This Page