Securing Your Server With Let's Encrypt

Discussion in 'Tips/Tricks/Mods' started by ahrasis, Feb 14, 2017.

  1. ahrasis

    ahrasis Member

    Securing Your Server With Let's Encrypt
    [Introduction]

    Earlier, I have posted on how to create Let's Encrypt for your server. However, there is a minor issue with ispserver.pem where it will need to be rebuilt manually upon Let's Encrypt renewal of its SSL files. This is due to the fact that it is not merely symlinked but rather created by combining several symlinks of Let's Encrypt files.

    It is pertinent to note that I am using Ubuntu 16.04 with Nginx, with Let's Encrypt SSL files creation and renewal from ISPConfig 3 fully working, so I do not guarantee this to work for all of you on other builds or distros. Certain modifications may be necessary to make it work for you. So let us start the guide once more time for the benefits of all.

    [Creating Your Server Website]
    1. Create a site for your server in ISPConfig panel via Sites > Website > Add new website. Remember! This is your server website and as such it must contain your server full domain name. I will refer to it as `hostname -f` in this guide, hopefully in can work without any changes for your server as well.

    [Accessing Your Server Online]
    2. Check if your server site is ready and accessible online. This is important because Let's Encrypt will need to verify your website is accessible online before issuing SSL key, certificate and chain file for your server site. Also note that you might or might not have to create dns for it. If you do need to create it, please do allow sometimes before its dns is properly propagated.

    [Securing Your Server With Let's Encrypt SSL]
    3. If your server site is already accessible via internet, go back to ISPConfig panel > Sites > Website > Website Name, then click SSL and Let's Encrypt check buttons and save. If successful, this will create Let's Encrypt SSL files and enable them for your server site. At this point only your server site is using this Let's Encrypt SSL files and not your ISPConfig page yet. If unsuccessful, you will not be able to proceed further, so do check its log file for a clue.

    [Activating Your Server ISPConfig SSL]
    4. Now, if you haven't enabled SSL during ISPConfig setup, enable it by typing ispconfig_update.sh in the terminal and select yes for SSL. We don't need this to be a proper key nor do we want to keep it but we want to work faster, thus we can simply enter for all of its fields. When you finished this, the self-signed SSL should already be enabled for your ISPConfig.

    [Checking Your Server ISPConfig SSL]
    5. Check your browser to confirm. Note that you might get some warning at this stage since the created SSL files are self-signed but the browser will confirm that your ISPConfig has SSL enabled or otherwise.

    [Using Your Server Let's Encrypt SSL For Your ISPConfig]
    6. If SSL is already enabled for your ISPConfig, then go to your server terminal, go root via sudo su and use the following command to backup and replace the created self-signed SSL files with Let's Encrypt SSL files. Note that we are using Let's Encrypt live folder instead of archive folder.
    Code:
    mv /usr/local/ispconfig/interface/ssl/ispserver.crt /usr/local/ispconfig/interface/ssl/ispserver.crt.bak
    mv /usr/local/ispconfig/interface/ssl/ispserver.key /usr/local/ispconfig/interface/ssl/ispserver.key.bak
    mv /usr/local/ispconfig/interface/ssl/ispserver.pem /usr/local/ispconfig/interface/ssl/ispserver.pem.bak
    ln -s /etc/letsencrypt/live/`hostname -f`/fullchain.pem /usr/local/ispconfig/interface/ssl/ispserver.crt
    ln -s /etc/letsencrypt/live/`hostname -f`/privkey.pem /usr/local/ispconfig/interface/ssl/ispserver.key
    cat /usr/local/ispconfig/interface/ssl/ispserver.{key,crt} > /usr/local/ispconfig/interface/ssl/ispserver.pem
    chmod 600 /usr/local/ispconfig/interface/ssl/ispserver.pem
    - Also note the last line where ispserver.pem is created by combining files, thuswise, it will not be automatically renewed by Let's Encrypt unlike other files which we merely symlinked them. We will deal with this in the last part of this guide.
    - Note also that you either type in `hostname -f` or server1.example.com as the result is the same because `hostname -f`is server1.example.com.

    [Using Your ISPConfig Let's Encrypt SSL For Others]
    7. As additional tips, based on Securing Your ISPConfig 3 Installation you may want to use symlink to ispserver.key or .crt or .pem instead of directly pointing your postfix, dovecot, courier, pure-FTPd and monit to Let's Encrypt SSL files. For dovecot, if it is already using postfix SSL files, it is safe for you to ignore it. In details you only need to do the followings:
    a. For postfix:
    Code:
    cd /etc/postfix
    mv smtpd.cert smtpd.cert_bak
    mv smtpd.key smtpd.key_bak
    ln -s /usr/local/ispconfig/interface/ssl/ispserver.crt smtpd.cert
    ln -s /usr/local/ispconfig/interface/ssl/ispserver.key smtpd.key
    service postfix restart
    
    b. For dovecot: (Note this shouldn't exist together with courier)
    Check if this code exist by using nano /etc/dovecot/dovecot.conf
    Code:
    [...]
    ssl_cert = </etc/postfix/smtpd.cert
    ssl_key = </etc/postfix/smtpd.key
    [...]
    Leave them as they are if they exist. Otherwise, fix them. In any event, run service dovecot restart.
    c. For courier: (Note this shouldn't exist together with dovecot)
    Code:
    mv /etc/courier/imapd.pem /etc/courier/imapd.pem.bak
    mv /etc/courier/pop3d.pem /etc/courier/pop3d.pem.bak
    ln -s /usr/local/ispconfig/interface/ssl/ispserver.pem /etc/courier/imapd.pem
    ln -s /usr/local/ispconfig/interface/ssl/ispserver.pem /etc/courier/pop3d.pem
    service courier-imap-ssl stop
    service courier-imap-ssl start
    service courier-pop-ssl stop
    service courier-pop-ssl start
    
    d. For pure-FTPd:
    Code:
    cd /etc/ssl/private/
    mv pure-ftpd.pem pure-ftpd.pem_bak
    ln -s /usr/local/ispconfig/interface/ssl/ispserver.pem pure-ftpd.pem
    chmod 600 pure-ftpd.pem
    service pure-ftpd-mysql restart
    e. For monit: (If you have it installed in your server)
    Code:
    nano /etc/monit/monitrc
    Add the above symlink to ispserver.pem we created for pure-ftpd in here as well:
    Code:
    [...]
      set httpd port 2812 and
        SSL ENABLE
        PEMFILE /etc/ssl/private/pure-ftpd.pem
        allow admin:'secretpassword'
    [...]
    And restart monit:
    Code:
    service monit restart

    [Set Auto Renewal For Your ISPConfig ispserver.pem]
    8. In this last step, which I haven't found in any guide so far, is the automatic update of ispserver.pem as earlier hinted. Currently, it have to be manually changed right after Let's Encrypt automatically renewed your server SSL files. To avoid overlooking this, you may want to install incron as suggested in the respective incron tutorial and create a script to automatically update your ispserver.pem file, as follows:
    a. Via terminal command, install incron, then create the script file and edit it using nano:
    Code:
    apt install -y incron
    nano /etc/init.d/le_ispc_pem.sh
    b. Add this in the le_ispc_pem.sh:
    Code:
    #!/bin/sh
    rm /usr/local/ispconfig/interface/ssl/ispserver.pem
    cat /usr/local/ispconfig/interface/ssl/ispserver.{key,crt} > /usr/local/ispconfig/interface/ssl/ispserver.pem
    chmod 600 /usr/local/ispconfig/interface/ssl/ispserver.pem
    service pure-ftpd-mysql restart
    service monit restart
    service postfix restart
    service dovecot restart
    service nginx restart
    - Some people do not install monit, so they can safely remove it. Do adjust the above script accordingly.
    c. We then make it executable, add root as allowed user for incrontab and then edit incrontab file:
    Code:
    chmod +x /etc/init.d/le_ispc_pem.sh
    echo "root" >> /etc/incron.allow
    incrontab -e
    d. We add this line in it incrontab:
    Code:
    /etc/letsencrypt/archive/yourserverdomain/ IN_MODIFY ./etc/init.d/le_ispc_pem.sh

    [Restarting Your System]
    I think that is about it for Securing Your Server With Let's Encrypt. You may want to restart your system afterwards.
    Code:
    service nginx restart
    Special note: Thank you @till, @florian030, @Thaddeus, @Turbanator, @Jesse Norell and others for their observations, feedbacks and suggestions that lead to the improvement of this guide.
     
    Last edited: Apr 20, 2017 at 7:19 AM
  2. Turbanator

    Turbanator Member HowtoForge Supporter

    This is great, thank you. I think your cron line has a typo. le_ispcon_pem.sh should be le_isp_pem.sh
     
  3. ahrasis

    ahrasis Member

    Yes. You are right. Thank you for the fix. I already updated the guide with your fix. Thank you again.
     
  4. Jesse Norell

    Jesse Norell Well-Known Member

    Clever, I like it.

    Doesn't this end up with multiple inotifywait processes watching the privkey1.pem file, with a new one starting every day? (Maybe it prints a notice that it's running and exits, I've not tried it yet.) Might need le_ispc_pem.sh to use a lock file or something?

    You also need to restart nginx, pure-ftpd and monit whenever the certificate changes, maybe make it run as part of the loop in le_ispc_pem.sh.
     
  5. ahrasis

    ahrasis Member

    Though I have also said the above, there is always room for improvement. Any suggestion for that?

    By the way, I am not so sure why we need to restart nginx, pure-ftpd and monit whenever the certificate changes. To me they are just using symlink to ISPC ispserver.pem and that symlink will remains after any changes. Care to explain why since the symlink name remains the same even after the renewal?
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    The services won't recognize and pick up a changed SSL cert without restart. An real world example: we use LE as cert authority on the git.sipconfig.org server, the server runs nginx and gitlab without a controlpanel, during initial installation we missed to add a restart of nginx after cert renewal which caused an expired SSL cert warning in the browser while the actual cert was up to date. Adding a nginx restart fixed that.
     
    ahrasis likes this.
  7. ahrasis

    ahrasis Member

    Thank you till and Jesse Norell for their observations, feedbacks and suggestions that lead to the improvement of this guide.

    I changed the script to be an init.d script, add new header to it as well as -m to inotifywait and and service nginx restart as thing to be done upon such changes. The reason for its change to an init.d script is due to the fact that inotifywait may stop on system reboot unless it is made a part of update-rc.d. With this change there will be no need of the tedious cron job.

    I am still not sure if we should restart pure-ftpd and monit services as well. Please do explain why it is needed, if you want it to be added as well.

    Thank you again.
     
  8. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Maybe we can add pre and post-hooks to the certbot-call. A post-hook could restart several services like mail and ftp. Just LE decided, if there is a need for restarting a services.
     
  9. ahrasis

    ahrasis Member

    Interesting idea florian. I'd prefer a hook to LE as well but that would require a proper knowledge and experience with its code of which I do not have. I will leave that to the expert hands to build one.
     
  10. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    If you want to try post-hooks:
    Download 900-letsencrypt.inc.php.gz and unzip the file to /usr/local/ispconfig/server/lib/classes/cron.d
    Create a post-hook-file /usr/local/ispconfig/server/scripts/le_post-hook.sh. I.e.
    Code:
    #!/bin/sh
    service postfix restart
    service dovecot restart
    and set the permissions:
    Code:
    chown root.root /usr/local/ispconfig/server/scripts/le_post-hook.sh
    chmod 700 /usr/local/ispconfig/server/scripts/le_post-hook.sh
    
    The post-hook will be called every time a cert is renewed. Maybe it's possible to limit the restarts to several domains.
     
  11. ahrasis

    ahrasis Member

    Great code. So, basically, we need to modify the default 900-letsencrypt.inc.php or replace it with the one you provide. This is creating one question - will it be replaced by an ISPConfig update?

    And if the postfix and dovecot restart is necessary, (because for now I still do not think they are yet, similar to pure-ftpd and monit), it would rather be easier if they are included in the suggested script in the above guide.
    Code:
    while inotifywait -m -q -e modify,attrib /etc/letsencrypt/archive/yourserverdomain/privkey1.pem >/dev/null; do
      rm /usr/local/ispconfig/interface/ssl/ispserver.pem
      cat /usr/local/ispconfig/interface/ssl/ispserver.{key,crt} > /usr/local/ispconfig/interface/ssl/ispserver.pem
      service nginx restart
      [ADD ANY EXTRA / NEEDED CODE IN HERE]
      [ADD ANY EXTRA / NEEDED CODE IN HERE]
      [ADD ANY EXTRA / NEEDED CODE IN HERE]
    done[code]
     
  12. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    An ispconfig-update will overwrite this. It's more like a quick hack but if this is working, i will add this to the main-code.
    With a post-hook you can trigger an action if you you get new cert. Your script checks only a specified cert. I think, it's easier to just define some hooks in a script. Especially, when you run a multiserer-setup with just one master and one interface. It should possible to copy new cerst over ssh with a hook-script to another server.
    You have to reload postfix and dovecot after you changed the cert. You may want to add the bundle to postfix and dovecot, too. I saw some problems with outlook when the bundle was not defined.
     
  13. ahrasis

    ahrasis Member

    Great. It would be nicer if by default, ISPConfig could create its own Let's Encrypt SSL, renew it and then restart all necessary services upon its Let's Encrypt SSL files renewal.

    Note that I did not include restart the system earlier because I was under the impression that ISPConfig should restart it after any Let's Encrypt SSL renewal. Then, after reading the replies, I realized that ispserver.pem may not be deleted and rebuilt at that time yet, so I added it in, just to make sure.

    However, I am still in doubt to add restart for other services though adding them will be more prudent. So in the above guide, I added restarting them as an option instead.

    By the way, what do you mean by bundle? If you mean the certificate and the chain, that are already included, because from what I understand, they are already bundled in the fullchain.pem. So in the above guide, I used:
    Code:
    ln -s /etc/letsencrypt/live/yourserverdomain/fullchain.pem /usr/local/ispconfig/interface/ssl/ispserver.crt
     
    Last edited: Feb 16, 2017
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    I doubt that any service will recognize a changed SSL cert without restart as these certs were normally read only once when the service starts and it is kept in memory then, so it is very likely that you have to restart each service that uses this SSL cert on renewal.
     
    ahrasis likes this.
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    Bundle is another term for chain certificate.
     
    ahrasis likes this.
  16. ahrasis

    ahrasis Member

    Thank you for the feedback. I edited the guide to for all them to be included and restarted upon Let's Encrypt SSL renewal (not as option anymore).

    So, I get that bundle as already included and correct. Thank you till.
     
    Last edited: Feb 16, 2017
  17. Jesse Norell

    Jesse Norell Well-Known Member

    As @till has indicated, anything that holds the certificate in memory probably needs restarted; pure-ftpd does for sure, I have no experience with monit.

    On the design of implementation from ispconfig triggers, I would probably not use inotifywait myself. Normally letsencrypt will renew the certificate with there is still a fair bit of time (I think 1 month?) until the old certificate expires - you definitely don't need up to the second service reloads after the certificate changes. As a standalone monitoring script I think it is clever and would have very quick response to certificate changes. For an ispconfig hook like @florian030 is suggesting I would compare the pure-ftpd pid file timestamp with the certificate timestamp, and only restart pure-ftpd if the certificate is newer.
     
  18. ahrasis

    ahrasis Member

    For me, I will not use the suggested replacement of 900-letsencrypt.inc.php until it is implemented by ISPConfig, just in case I need to update ISPConfig as it might wipe out the replacement.

    And on the contrary, I do believe inotifywait will check and only acted upon a change of the specified file and this includes the attributes. So it is basically safer and more practical for the time being.
     
  19. Tuumke

    Tuumke Member

    I use ISPConfig for personal websites only at the moment.
    How would use my mail setup so that it sees the letsencrypt certs?
    Currently each domain has its own A 3600 mail.domain.tld pointing to the server ip.
    The MX record points to the A 3600 mail.domain.tld

    Do i have to setup the auto-subdomain to * or something so that the ssl cert is valid for domain.tld and *.domain.tld?
     
  20. Jesse Norell

    Jesse Norell Well-Known Member

    ISPConfig currently only requests letsencrypt certificates for websites, so use with email must be setup manually for now. The instructions in this thread only request a certificate for the server's full hostname (output of `hostname -f`); you can request a certificate which includes multiple names (server hostname, mail.domain1.com, mail.domain2.com, etc.) from the command prompt, which would work if you have a single server installation (not multi-server). You would need to request a new certificate from the command line every time you add or remove a name from it; as long as the first name in the certificate is `hostname -f`, then the instructions above to configure postfix and dovecot should work (ie. point to the certificate/key files in /etc/letsencrypt/live/`hostname -f`/).

    Note you can only have I believe 100 names included in a letsencrypt certificate, so doing this is only practical for small sites/config. Note also that if any of the domains you have move away from your server, so that mail.thatdomain.com no longer resolves to your server's ip address, then you must request another certificate without that name included; if you do not, the certificate cannot be renewed as it is, and will expire in < 90 days.

    Another option is to use a different letsencrypt client, acme.sh, and this would work in a multi-server installation. It's a little more complicated to setup, but not too bad; @sjau has posted a howto and various other info on it on the forums here. It sounds like IPSConfig may switch to using acme.sh in the future (or maybe support both clients).

    If interested, our current solution to this is to tell customer they can use mail.theirdomain.com as the server name for convenience, but they will get certificate errors; or they can use the actual server hostname without errors, but will probably have to change that every now and then (eg. every few years) whenever their domain moves to a new server. Some pick/prefer one, some the other, and I think most don't care.
     

Share This Page