ISPConfig - Let's Encrypt errors.

Discussion in 'Installation/Configuration' started by Bookworm, Mar 6, 2019.

  1. Bookworm

    Bookworm Member

    I've spent several hours debugging, and I've tracked down a problem with ISPConfig's handling of vhosts with LetsEncrypt.

    Specifically, if a certificate _already exists_, ISPConfig will refuse to modify the vhost file. It will only modify the vhost file if it can issue its own certificate. This is a problem if you already _have_ a cert, and had to rework a site for some reason. If you're not ready for renewal, ISPConfig spits on you.

    Here's a clip from the log, plus what happened when I ran the command manually. The vhost in question is marked as having SSL enabled, but not Let's Encrypt - despite there being no actual errors. There is no SSL section in the vhost.

    Is this normal behaviour, and is there a way to work around it?

    06.03.2019-06:08 - DEBUG - Create Let's Encrypt SSL Cert for:
    06.03.2019-06:08 - DEBUG - Let's Encrypt SSL Cert domains: --domains --domains
    06.03.2019-06:08 - DEBUG - exec: /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v02.api.let --rsa-key-size 4096 --email [email protected] --domains --domains --webroot-path /usr/local
    06.03.2019-06:08 - DEBUG - Let's Encrypt Cert file: does not exist.
    06.03.2019-06:08 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/
    06.03.2019-06:08 - DEBUG - Apache status is: running
    06.03.2019-06:08 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    06.03.2019-06:08 - DEBUG - Restarting httpd: systemctl restart apache2.service
    06.03.2019-06:08 - DEBUG - Apache restart return value is: 0
    06.03.2019-06:08 - DEBUG - Apache online status after restart is: running
    06.03.2019-06:08 - DEBUG - Processed datalog_id 6452

    [email protected]:/usr/local/ispconfig/server# /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server --rsa-key-size 4096 --email [email protected] --domains --domains --webroot-path /usr/local/ispconfig/interface/acme
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Cert not yet due for renewal
    Keeping the existing certificate

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Certificate not yet due for renewal; no action taken.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    (Edited - deleted out the failed paste command line. )
    Last edited: Mar 6, 2019
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    This might be related to this:

    The question is how you created that LE cert that already exists, did you use the exact same command that ISPconfig is using, or did you use a different method? E.g. when you would have used certbot with apache option, then certbot tries to create a copy of the vhost file which makes it impossible for ispconfig to further manage that website due to the duplicate manual config. You might want to check the apache sites-available and sites-enabled config directories to see if there are any config files with -le in their name, if that's the case, then you must have run certbot manually with apache plugin and this broke your setup.
  3. Bookworm

    Bookworm Member

    Even more cute. I purged _all_ of my Let's Encrypt certs, and ran the utility. It happily created the certificate, then wrote a vhost file that still contains nothing to do with SSL.
  4. Bookworm

    Bookworm Member

    I used the _exact_ command that ISPConfig displayed in the debug logs - as you can see in the first posting.

    I had run certbot manually in the past, but all of those files were purged.
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Really strange. When the debug log says:

    06.03.2019-06:08 - DEBUG - Let's Encrypt Cert file: does not exist.

    then ISPConfig did not found the cert and that's why SSL is not activated. The question is why it did not found the cert, certs are not selected by their file name (as they can change from time to time), the cert is selected based on the renew config files.

    Please post the renew config file from /etc/letsencrypt/renewal/ folder of this domain. Compare the domains listed in there if they match the domains of the site and ensure that all subdomains are listed in there too.
  6. Bookworm

    Bookworm Member

    # renew_before_expiry = 30 days
    version = 0.31.0
    archive_dir = /etc/letsencrypt/archive/
    cert = /etc/letsencrypt/live/
    privkey = /etc/letsencrypt/live/
    chain = /etc/letsencrypt/live/
    fullchain = /etc/letsencrypt/live/

    # Options used in the renewal process
    account = 945e6da4eb0490598b05570f4d486ad1
    server =
    authenticator = webroot
    rsa_key_size = 4096
    webroot_path = /usr/local/ispconfig/interface/acme,
  7. Bookworm

    Bookworm Member

    It still keeps repeating that the file doesn't exist - when it clearly does. I've deleted it twice, and it happily recreates it, then claims that the cert doesn't exist.

    So, something's broken. Could you tell me how to bypass the cert check and force it to write the SSL changes? I'm certain I can find it myself, but it could take several hours.
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    and there is really nothing after [[webroot_map]] line in that file?

    Here is a file from my server, as you can see, it lists the web roots:

    # renew_before_expiry = 30 days
    version = 0.21.1
    archive_dir = /etc/letsencrypt/archive/domain.tld
    cert = /etc/letsencrypt/live/domain.tld/cert.pem
    privkey = /etc/letsencrypt/live/domain.tld/privkey.pem
    chain = /etc/letsencrypt/live/domain.tld/chain.pem
    fullchain = /etc/letsencrypt/live/domain.tld/fullchain.pem
    # Options used in the renewal process
    account = xxxxxxxxxxxxxxxxxxxxxx
    authenticator = webroot
    rsa_key_size = 4096
    installer = None
    domain.tld = /usr/local/ispconfig/interface/acme
    www.domain.tld = /usr/local/ispconfig/interface/acme
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    The cert check is in /usr/local/ispconfig/server/lib/classes/
  10. Bookworm

    Bookworm Member

    Nope, that's it.

    All of the files in that directory end at [[webroot_map]]

    (There are four. I have some manual certs for postfix, etc. )
  11. Bookworm

    Bookworm Member

    I need to build a new server and manually migrate all of my hosted sites to it; I guess this drives home that I _have_ to do that ASAP. I just hate doing all that work.
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Instead of trying to change the ispconfig code, alter the certbot config files and add the missing web root paths in the way it is shown in my example file and then tick the le and ssl checkboxes in the site again and see how it goes. We will try to find out why certbot fails to add the domains in its config file, maybe its a bug in a specific certbot version or certbot stores this info somwhere else now.
  13. Bookworm

    Bookworm Member

    Giving that a shot.
  14. Bookworm

    Bookworm Member

    That forced ISPconfig to realize the cert was there, and that site is now functioning with SSL.

    I'll do the other two/three I need to have functioning, and finish my other work for the night.
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    The issue is really strange, two ISPConfig developers tested this now independently on different servers, installed latest certbot 0.31 and the current ISPConfig version 3.1.13p1 and when we create a new SSL cert, all domains that belong to that cert are listed in the
    [[webroot_map]] section of the newly generated cert.
  16. Bookworm

    Bookworm Member

    Even with an old cert (0.23) that was created as standalone, once I added the webroot_map section, ispconfig would use it. Why does ISPConfig refuse to use certs maintained differently? Just curious, now.

    I'm not going to suggest putting a lot of effort into it. I'm going to build a new server and work on gradually merging my two servers into one, then creating a second just to be a slave for DNS.
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    The problem is that certbot tends to rename certs, adds suffixes to cert files and also to use other domains which are in the cert as file name. So you can not assume that just because the cert is named domain.tld.crt that this belongs to a website domain.tld.crt, it can be for a different site or different subset of domains, certbot has a long history of such issues. That's why ispconfig reads in all cert config files and checks all the domains inside against the domains, subdomains and alias domains and only if that matches, the cert will be used. If you created a cert manually with a wrong method and this results in certbot to fail to write the webroot map, then this cert is excluded from being used as it basically contains no usable domains.

    I explained it many many times to NEVER create LE certs manually on the shell on an ISPConfig system if you plan to have a site with the same name as this cert will not get used and you will not be able to use SSL in a website with that domain anymore in ISPConfig.
  18. Bookworm

    Bookworm Member

    Well - the two certs I was trying to use _weren't_ created manually. I did find a glitch, in that I had to use a newer version of certbot, to get ISPConfig to recognize that it was even installed (0.23 is apparently too old) - but the instructions for using/installing Certbot/LetsEncrypt also need to be updated, because the options mentioned in the post don't match up with what certbot-auto does anymore.

    I've noticed that in the last few years, as Certbot and LE have changed - they REALLY change stuff very version, rather than only changing what's necessary. It's almost as bad as Microsoft or Firefox. "Hey, let's change it because our marketing people say it's going to be great!" Yeesh.
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    The way you install certbot does not matter, you can install it from packages of your Linux distribution or with certbot-auto program, ISPConfig recognizes it in all common places, even the original letsencrypt client is recognized and used if no certbot is installed. As you can see in my post above, the first cert was created with certbot 0.21 with ispconfig, so the version was even older than yours and worked fine with ISPConfig.

    Why certbot on your server failed to add a webroot map when not executed manually, I don't know. It is not reproducible here, tested by 3 developers on 3 different servers now.
  20. Bookworm

    Bookworm Member

    That's why I said it's not worth chasing. My server is ... rather old now. Well, the hardware has been upgraded, but ISPConfig has been upgraded and updated over at least one full major revision, plus multiple minor revisions of both, plus OS upgrades. So, there's probably stuff scattered in my ISPConfig that doesn't match anyone else's.
    till likes this.

Share This Page