ISPCONFIG 3.1.13 + UBUNTU 16.04 + CERTBOT 0.26 upgraded to 0.28 do Apache2 do not start anymore

Discussion in 'Installation/Configuration' started by Fabio IT Consultant, Jan 28, 2019.

  1. Hello
    Today i've upgraded certbot from version .026 to 0.28 to accomplish compatibility to deprecated TLS-SNI-01 to be replaced by HTTP ou DNS validation....
    After that apache2 do no restart anymore...
    After trying many things i can't find a way to get it running correctly anymore.
    every time that i try to restart apache2 this error come up:
    Logs
    /var/log/apache2/error.log:

    [Mon Jan 28 16:28:28.004671 2019] [ssl:emerg] [pid 14371] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/ispconfig/httpd/mydomainXXX.com/error.log for more information
    AH00016: Configuration Failed
    /var/log/ispconfig/httpd/mydomainXXX.com/error.log:
    [Mon Jan 28 16:28:28.004419 2019] [ssl:emerg] [pid 14371] AH02572: Failed to configure at least one certificate and key for mydomainXXX.com:443
    [Mon Jan 28 16:28:28.004588 2019] [ssl:emerg] [pid 14371] SSL Library Error: error:0906D06C:pEM routines:pEM_read_bio:no start line (Expecting: DH PARAMETERS) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile?
    [Mon Jan 28 16:28:28.004614 2019] [ssl:emerg] [pid 14371] SSL Library Error: error:0906D06C:pEM routines:pEM_read_bio:no start line (Expecting: EC PARAMETERS) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile?
    [Mon Jan 28 16:28:28.004658 2019] [ssl:emerg] [pid 14371] SSL Library Error: error:140A80B1:SSL routines:SSL_CTX_check_private_key:no certificate assigned

    Versions installed in the Ubuntu 16 server thats is regularly updated:
    Server version: Apache/2.4.18 (Ubuntu)
    Server built: 2018-06-07T19:43:03

    Certbot: 0.28
    ISPConfig 3.1.13

    Apache do not start anymore showing this error:
    [email protected]:~# apache2ctl graceful
    httpd not running, trying to start
    Action 'graceful' failed.
    The Apache error log may have more information.
     
    Last edited: Jan 29, 2019
  2. nhybgtvfr

    nhybgtvfr Active Member

    dunno the answer to your problem, but I did post a question about the cerbot validation changes, and have been told cerbot with ispconfig doesn't use tls-sni-01, so that shouldn't be a problem. no idea whether using anything before certbot 0.28 will be a problem, (I currently have 0.23), i'm prepared to wait till feb 13/14 to find out since the certs should still work anyway.
    so you may be able to revert your certbot version.

    only other thing I can think of currently is maybe a newer version of openssl is required?
     
    Fabio IT Consultant likes this.
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    Maybe you accidentally removed certificate files from /etc/letsencrypt/ folder?
     
  4. Tim
    Pretty much honestly, i've made everything that an well experienced professional would do.
    But the problem has begun after done the following instructions: from https://community.letsencrypt.org/t/how-to-stop-using-tls-sni-01-with-certbot/83210
    ""
    Let’s Encrypt is removing support for domain validation with TLS-SNI-01. If you’re using Certbot and received an email titled “Action required: Let’s Encrypt certificate renewals”, here’s how to fix the problem. It’s possible you’ve upgraded Certbot in the time since the last TLS-SNI validation mentioned in the email, in which case you’re fine. These instructions tell you how to check.

    1. Confirm your Certbot version is 0.28 or higher:

      certbot --version || /path/to/certbot-auto --version
    If the version is less than 0.28, you need to upgrade your Certbot. Visit https://certbot.eff.org/ 5.9k and follow the instructions for your webserver and OS.

    1. Remove any explicit references to tls-sni-01 in your renewal configuration:

      sudo sh -c "sed -i.bak -e 's/^\(pref_challs.*\)tls-sni-01\(.*\)/\1http-01\2/g' /etc/letsencrypt/renewal/*; rm -f /etc/letsencrypt/renewal/*.bak"

    2. Do a full renewal dry run:

      sudo certbot renew --dry-run
    If the dry run succeeds, and your Certbot version is 0.28 or higher, you’re good to go! If it fails, fix the validation problems you see and try again.

    If you get a connection refused or connection timeout, you may have a firewall blocking port 80. tls-sni-01 used port 443, but http-01 uses port 80. Ideally your web server should allow both ports. If that’s not possible, for instance because your ISP blocks port 80, you’ll need to switch to the dns-01 challenge, or use an ACME client that supports tls-alpn-01.

    Note: if you installed Certbot in late 2015 or early 2016, it may be called letsencrypt or letsencrypt-auto (the project was renamed). Follow the instructions at https://certbot.eff.org 2.0k to install the latest version.

    Credit to @_az for the suggestion to write more step-by-step instructions and @jsha for rewriting these instructions with that suggestion in mind.


    We done the update version above once we received the email below from [email protected]
    "Hello,
    Action may be required to prevent your Let's Encrypt certificate renewals
    from breaking.
    If you already received a similar e-mail, this one contains updated
    information.
    Your Let's Encrypt client used ACME TLS-SNI-01 domain validation to issue
    a certificate in the past 60 days. Below is a list of names and IP
    addresses validated (max of one per account):
    mydomain.com.br (xxx.yyy.zzz.uuu) on 2019-01-09
    TLS-SNI-01 validation is reaching end-of-life. It will stop working
    temporarily on February 13th, 2019, and permanently on March 13th, 2019.
    Any certificates issued before then will continue to work for 90 days
    after their issuance date.

    You need to update your ACME client to use an alternative validation
    method (HTTP-01, DNS-01 or TLS-ALPN-01) before this date or your
    certificate renewals will break and existing certificates will start to
    expire.

    Our staging environment already has TLS-SNI-01 disabled, so if you'd like
    to test whether your system will work after February 13, you can run
    against staging: https://letsencrypt.org/docs/staging-environment/

    If you're a Certbot user, you can find more information here:

    https://community.letsencrypt.org/t/how-to-stop-using-tls-sni-01-with-certbot/83210

    Our forum has many threads on this topic. Please search to see if your
    question has been answered, then open a new thread if it has not:

    https://community.letsencrypt.org/

    For more information about the TLS-SNI-01 end-of-life please see our API
    announcement:

    https://community.letsencrypt.org/t/february-13-2019-end-of-life-for-all-tls-sni-01-validation-support/74209

    Thank you,
    Let's Encrypt Staff
    "
     
  5. Once i follow the best pratices that i've spread out among many IT Technicians that worked in my teams, and also once i've always enforced and used deeply ITIL since 1999 working for UK internacional companies...i fortunately had backups from /etc and a full backup of my server...
    So to uncommit the changes, i've recover the /etc tar.gz and renamed at /etc/apache2/sites-available the mydomain.vhost file and after that the apache2 came up back, finally.
    I've realized that certbot 0.28 goes intro the vhost files from the domains that i had renewed the certificates directly from certbot cli commands, and this new version changed the paths certificates from /www/clients.... that are in fact ln links to path directly to /etc/letsencrypt/live/xxxdomais certificates...
    I've deactivated the sites and turned off SSL and Letsencrypt by unchecking the regarding checkboxes of those domains...after that, reactivated and renabled the SSL and Letsencrypt checkboxes....to keep the domains on complaince to ISPConfig vhost format again, what i really prefer, to do not disrupte ISPConfig features and future updates or changes by the ispconfig panel, that are a real benefit from what you guys perfectly done....ISPConfig is really fabulous....i love it.
    So now i have a few trick questions:
    1) Should i go back to previous version 0.26 or can i keep the version 0.28?
    If i keep using 0.28 the renew process always produce an alert line in red color saying that:
    Obtaining a new certificate
    Performing the following challenges:
    tls-sni-01 challenge for xxxxxxx.com.br
    TLS-SNI-01 is deprecated, and will stop working soon. (THIS LINE IS RED COLOR IN THE TERMINAL)
    Waiting for verification...
    Cleaning up challenges
    Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/XXXX.com.br​
    2) Should i change letsencrypt to do not use TLS-SNI-01 challenge anymore or do nothing?
    3) What will happen at February 13th?
    Comment: The letsencrypt warning email said ""
    TLS-SNI-01 validation is reaching end-of-life. It will stop working
    temporarily on February 13th, 2019, and permanently on March 13th, 2019.
    Any certificates issued before then will continue to work for 90 days
    after their issuance date."
    4) ISPConfig renew crontab script suggestions will keep running automatically considering the conditions above?
    5) Curiously i have many sites running with path and certificates generated by ISP panel as also have other sites running well too with certificates using /etc/letsencrypt path and certificates renewed by certbot. OH My GOD...Linux are really clever!
    6) Last one: Why in /etc/apache2/sites-available there is some files with both xxxdomain.com.br { .vhost and .conf} and others ones with just xxxdomain.com.br.vhost? What should really be there, both or just .vhost files?

    Thanks in advance to you Till and our forum contributors.
     
    Last edited: Jan 29, 2019
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Let's encrypt provides step by step instructions that show you how to proceed when tls-sni-01 is used on your server: https://community.letsencrypt.org/t/how-to-stop-using-tls-sni-01-with-certbot/83210
    There is nothing ispconfig specific here, ispconfig runs certbot in the exact same way as you would run it on servers without ispconfig, ispconfig does not use the apache mode though as certbot shall never try to edit apache config files on its own, so only the cert is requested.
     
    Fabio IT Consultant likes this.
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    What I wonder if this line "Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/XXXX.com.br " is shown by certbot or ispconfig? Certbot should not try to modify any vhosts and it does not do that when you enable SSL on an ispconfig server trough ISPConfig, if certbot would try to modify a vhost file, then your server would break as certbot makes some mistakes when it edits files. You should check the sites-available and sites-enabled folder from apache, it should not contain any vhost files with "-le" in the file name. If there are such files, then probably certbot was issued manually on the shell with apache option.
     
    Fabio IT Consultant likes this.
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    I've just rethought your posts and I guess the problem you encountered is most likely caused by the manual renewal of certs that you did on cli, if I understood you correctly.
     
    Fabio IT Consultant likes this.
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Fabio IT Consultant likes this.
  10. i would like to keep using the ISPConfig method that you said is webroot...where should i change it to do not use TNS anymore, something that i've bet that came up after certbot updated to version 0.28.
     
  11. understood, that text message is from certbot cli command to renew certificates....but increadibly both methods are running on your perfect server Till... you guys are really clever...Congratulations...this ISPConfig can solve things that are not even predicted by you guys.
     
  12. This instructions caused the problem indeed...lol...i really do not want to use those instructions in fact...otherwise the problem will come up again...if just in case you agree with my conclusion and perspective about it...lol
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    I would use the sed command that the LE team posted:

    sh -c "sed -i.bak -e 's/^\(pref_challs.*\)tls-sni-01\(.*\)/\1http-01\2/g' /etc/letsencrypt/renewal/*; rm -f /etc/letsencrypt/renewal/*.bak"

    I wonder why you got tls-sni-01, I don't have that method of any of my servers, all have http-01 without the need of changing anything now.

    I just made a short test with 'certbot renew --dry-run' command on ym server where I have certbot 0.30.2 at the moment, it showed a line:

    Plugins selected: Authenticator webroot, Installer None

    which looks fine to me as Installer None probably means that it will not try to edit any config files.

    It depends on the php modes that you use. If you would manually use certbot on a vhost with php-fpm enabled, then certbot will duplicate this vhost but it does not recognize that the php-fpm identifier must be unique for the whole server, the result then is that apache will fail to start due to duplicate identifiers used in its config. That's why one should not use certbot manually in a mode that modifies the apache config, its parser will fail on more complex vhost files.
     
    Fabio IT Consultant likes this.
  14. "I wonder why you got tls-sni-01, I don't have ...."
    I am quite sure that the TNS-SL-01 came up as effect of following instructions or after i've faced any error, some forum suggested to goes to letsencrypt.conf or directory /etc/letsencrypt/renewal WHEN I'VE INSTALLED THE SERVER...but i use ISPCONFIG for more than 4 years...
    Looking into two different domain .conf files that i have in /etc/letsencrypt/renewal, what is the best pratice or the correct format to be 100% compliant to ISP:
    1) "# renew_before_expiry = 30 days
    version = 0.28.0
    archive_dir = /etc/letsencrypt/archive/vps.xxxdomain.com.br
    cert = /etc/letsencrypt/live/vps.xxxdomain.com.br/cert.pem
    privkey = /etc/letsencrypt/live/vps.xxxdomain.com.br/privkey.pem
    chain = /etc/letsencrypt/live/vps.xxxdomain.com.br/chain.pem
    fullchain = /etc/letsencrypt/live/vps.xxxdomain.com.br/fullchain.pem

    # Options used in the renewal process
    [renewalparams]
    server = https://acme-v02.api.letsencrypt.org/directory
    account = xxxxxxxxyyyyy
    authenticator = webroot
    [[webroot_map]]
    vps.xxxdomain.com.br = /var/www/html
    "
    or

    2)
    # renew_before_expiry = 30 days
    version = 0.28.0
    archive_dir = /etc/letsencrypt/archive/yyydomain.com.br
    cert = /etc/letsencrypt/live/yyydomain.com.br/cert.pem
    privkey = /etc/letsencrypt/live/yyydomain.com.br/privkey.pem
    chain = /etc/letsencrypt/live/yyydomain.com.br/chain.pem
    fullchain = /etc/letsencrypt/live/yyydomain.com.br/fullchain.pem

    # Options used in the renewal process
    [renewalparams]
    server = https://acme-v02.api.letsencrypt.org/directory
    authenticator = apache
    account = yyyyyyyyyyyyyy
    installer = apache

    or perhaps a third format LIKE I HAVE ALSO...ps: I've never change it....i do not have the habitual to change standard things...i know thta it can cause real mess.
    3)
    # renew_before_expiry = 30 days
    version = 0.26.1
    archive_dir = /etc/letsencrypt/archive/zzzdomain.com.br
    cert = /etc/letsencrypt/live/zzzdomain.com.br/cert.pem
    privkey = /etc/letsencrypt/live/zzzdomain.com.br/privkey.pem
    chain = /etc/letsencrypt/live/zzzdomain.com.br/chain.pem
    fullchain = /etc/letsencrypt/live/zzzdomain.com.br/fullchain.pem

    # Options used in the renewal process
    [renewalparams]
    account = zzzzzzzzzzzzzzz
    server = https://acme-v01.api.letsencrypt.org/directory
    authenticator = webroot
    rsa_key_size = 4096
    post_hook = echo '1' > /usr/local/ispconfig/server/le.restart
    [[webroot_map]]
    zzzdomain.com.br = /usr/local/ispconfig/interface/acme
    www.zzzdomain.com.br = /usr/local/ispconfig/interface/acme
     
  15. In fact i have domains using different PHP methods, fastcgi, hhvm, etc...it explains why it is happening, many thanks Till...
     
  16. i really researching the web to find instructions to disable TLS-SN-01 and return to http challenge method.....but just in case you know where to change it, please let me know...
    Thanks a lot in advance.
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Files which contain:

    authenticator = apache
    installer = apache

    must be from a manual certbot run, so I guess you must edit these to use authenticator webroot and probably remove the installer line.
    also a "post_hook = echo '1' > /usr/local/ispconfig/server/le.restart" line is good as this ensures that apache will be restarted correctly after renewal.

    The sed command should be fine. Your problem arises most likely due to the manual use of certbot which resultet in renewal files that started to modify apache configs.

    if server 01 or 02 is used should not matter.
     
    Fabio IT Consultant likes this.
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    And in webroot_map, all domains should point to /usr/local/ispconfig/interface/acme, so file 3 is a file created by ispconfig.
     
    Fabio IT Consultant likes this.
  19. I've already changed it as you said.....pointing all domains to ....../acme
    added posthook also
    and commented installer line
    and changed authenticator = webroot for the domains that had "apache"
     
  20. Morning Till
    Could you please read the questions above and reply to me those that you could contribute with your thought about...
     

Share This Page