Outdated LE On Ova

Discussion in 'ISPConfig 3 Priority Support' started by yupthatguy, Jun 7, 2021.

  1. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Morning Fellas,

    It appears that the current LE certbot that is installed on the .ova file is outdated. Under Monitoring -> LE log I get the following warning:

    Code:
    2021-06-06 09:01:07,657:DEBUG:certbot.main:certbot version: 0.35.1
    2021-06-06 09:01:07,657:DEBUG:certbot.main:Arguments: ['-n', '--post-hook', "echo '1' > /usr/local/ispconfig/server/le.restart"]
    2021-06-06 09:01:07,657:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2021-06-06 09:01:07,663:WARNING:certbot.cli:You are running with an old copy of letsencrypt-auto that does not receive updates, and is less reliable than more recent versions. The letsencrypt client has also been renamed to Certbot. We recommend upgrading to the latest certbot-auto script, or using native OS packages.
    2021-06-06 09:01:07,664:DEBUG:certbot.cli:Deprecation warning circumstances: /opt/eff.org/certbot/venv/bin/certbot / {'LANG': 'en_US.UTF-8', 'SHELL': '/bin/sh', 'LANGUAGE': 'en_US:en', 'SHLVL': '1', 'OLDPWD': '/root', 'PWD': '/usr/local/ispconfig/server', 'LOGNAME': 'root', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin', 'HOME': '/root', '_': '/bin/php'}
    2021-06-06 09:01:07,684:DEBUG:certbot.log:Root logging level set at 20
    2021-06-06 09:01:07,685:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
    2021-06-06 09:01:07,694:DEBUG:certbot.renewal:no renewal failures
    
    What are the steps to correct this?
    Thx.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

  3. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Any ramifications from using an alternative method install LE?

    In an effort to keep my prod & test in sync, I tried the following that seems to work:

    Then, I did referenced the original The Perfect Server instructions that I followed, to install the same acme LE and force updated ispconfig to issue new self-signed certificate .

    Code:
    1.)#curl https://get.acme.sh | sh -s
    2.)#ispconfig_update.sh --force
    3.)#shutdown -r now
    
    Everything seems to be running perfectly fine... no errors or warnings anyway, -except-, I am a little worried because the Monitoring -> LE.log doesn't reflect any changes being made (still shows original warning, even after 5 minute refresh)

    Is there a a cron job (or something else) I should run confirm all is well?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    You switched from certbot to acme.sh as LE client, that's something which I can't recommend on any setup that is in use already as they are incompatible with each other, existing LE certs will fail and likely websites that use them will fail too sooner or later. That's why I suggested installing a new certbot version and not acme.sh.
     
  5. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Appreciate the feedback.

    As it stands, ispconfig_update.sh successfully issued a new certificate and previously, I haven't issued any LE ssl to any sites on my test server (actually no sites are installed, only nextcloud, which I didn't issue an ssl certificate to either.

    Will adding completely new sites, somehow be at risk as well? This is a vbox test server, so I am not terribly concerned about ssl, more worried about unforeseen bugs / errors cropping up.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    As long as the server has not been used yet, then it#s ok to switch to acme.sh. Jut don't do that on systems with active sites that use LE certs.
     
  7. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Does it show the certbot or acme log? It could be that if letsencrypt.log exists that's what is shown, and you would need to delete it for acme.log content to appear.
     
  8. yupthatguy

    yupthatguy Member HowtoForge Supporter

    @jesse-Norell Thanks for the tip. However, last night, after a 2nd reboot.. the test server suddenly "freaked out with NC not loading and ISPConfig showing blank pages. After some investigation, I may need to use @till's original suggestion of updating only the cert-bot, if the problem is related to the new SSL. However, there is also a chance the sudden failure could be related to a modsec vs. ssl issue. Even though the modsec_audit.log is clear of errors... the apache2 error.log shows some modsec related issues that need to be resolved. After, I resolve the modsec issues, I will test the SSL "shortcut" solution, I had and see how it goes know..
     
  9. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Upon closer examination of the apache error log, there was no modsec vs. ssl issue (forgive the bias, modsec is usually the problem). The errors that were in my log were caused by me using self-signed SSL certificates on my test server with the "SSLUseStapling" directive turned on in my vhost. I set the directives to "Off" and that got rid of my errors and got LE log file going again. I did this with an earlier vbox snapshot, so now I have the old certbot working correctly.

    From this point, I attempted @till's original recommendation and ran into my current problem.

    One more, note (symptom), perhaps unrlated... even though LE log is going again (no ssl errors)... I am unable to issue a self-signed certificate for my nextcloud install. I check the boxes, click save, I see the red save indicator... test in browser.. no https... I re-check ispconfig domain config, and the SSL boxes are NOT checked.. (reverted)...

    The official certbot instructions require me to install snapd (which I was trying to avoid with my acme.sh workaround). However, trying to do things the correct way, I have the problem that I can't -completely- install snapd on (debian 10 buster, inside virtualbox, oracle virtualization):

    On Step 3 of the official instructions, I attempt to run:
    https://certbot.eff.org/lets-encrypt/debianbuster-apache

    Code:
    #snap install core; sudo snap refresh core
    
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-389560980: unknown filesystem type 'squashfs'.
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-389560980: unknown filesystem type 'squashfs'.
    
    
    I tried the solutions that I found online:

    Code:
    #apt install libsquashfuse0 squashfuse fuse
    then
    # snap install core; snap refresh core
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-389560980: unknown filesystem type 'squashfs'.
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-389560980: unknown filesystem type 'squashfs'.
    
    No Luck.

    At which point, I had to idea to skip over the above, and just try Step Five of the official instructions:
    Code:
    #snap install --classic certbot
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-745794367: unknown filesystem type 'squashfs'.
    
    So, this appears to be an impasse, no way to actually upgrade certbot.. Anyone have any thoughts?

    FYI, another symptom / ssl problem. I can't issue a self-signed certificate for my nextcloud installation. I check the SSL boxes for the website -> domain , click save, wait for the red save to disapper. Then I check the browser.. no https. From here, I re-check ispconfig websites -> domain and the SSL boxes are no longer checked.:(
     
    Last edited: Jun 8, 2021
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    No issues with installing snap and updating certbot in the VM, just tested it by downloading the VM and updating snap. Probably you did some further modifications which cause you issues now.

    Code:
    [email protected]:~# mv /opt/eff.org /opt/eff.org_bak
    [email protected]:~# rm -f /usr/local/bin/certbot-auto
    [email protected]:~# apt install snapd
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following package was automatically installed and is no longer required:
      linux-image-4.19.0-4-amd64
    Use 'apt autoremove' to remove it.
    The following additional packages will be installed:
      squashfs-tools
    Suggested packages:
      zenity | kdialog
    The following NEW packages will be installed:
      snapd squashfs-tools
    0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
    Need to get 14.4 MB of archives.
    After this operation, 61.3 MB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 http://deb.debian.org/debian buster/main amd64 squashfs-tools amd64 1:4.3-12 [125 kB]
    Get:2 http://deb.debian.org/debian buster/main amd64 snapd amd64 2.37.4-1+b1 [14.3 MB]
    Fetched 14.4 MB in 2min 21s (102 kB/s)
    Selecting previously unselected package squashfs-tools.
    (Reading database ... 71070 files and directories currently installed.)
    Preparing to unpack .../squashfs-tools_1%3a4.3-12_amd64.deb ...
    Unpacking squashfs-tools (1:4.3-12) ...
    Selecting previously unselected package snapd.
    Preparing to unpack .../snapd_2.37.4-1+b1_amd64.deb ...
    Unpacking snapd (2.37.4-1+b1) ...
    Setting up squashfs-tools (1:4.3-12) ...
    Setting up snapd (2.37.4-1+b1) ...
    Created symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
    Created symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
    Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /lib/systemd/system/snapd.service.
    Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /lib/systemd/system/snapd.socket.
    Processing triggers for man-db (2.8.5-2) ...
    Processing triggers for mime-support (3.62) ...
    [email protected]:~# snap install core
    2021-06-08T09:24:25+02:00 INFO Waiting for restart...
    core 16-2.50.1 from Canonical✓ installed
    Channel latest/stable for core is closed; temporarily forwarding to stable.
    [email protected]:~# snap refresh core
    snap "core" has no updates available
    [email protected]:~# snap install --classic certbot
    Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
             installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
             details.
    
    certbot 1.16.0 from Certbot Project (certbot-eff✓) installed
    [email protected]:~# ln -s /snap/bin/certbot /usr/bin/certbot
    [email protected]:~# certbot --version
    certbot 1.16.0
    [email protected]:~#
    Btw. Deleting certbot-auto is not needed to make this work, I just deleted it to clean things up. I checked for certbot and letsencrypt binaries or symlinks on the VM first, that's why I could skip the 4 rm commands I mentioned in earlier posts.
     
  11. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Are you using oracle virtualbox?
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

  13. yupthatguy

    yupthatguy Member HowtoForge Supporter

    No customizations here... snap just doesn't like oracle virtualizations

    Found this:
    https://community.letsencrypt.org/t...-mount-squashfs-image-using-squashfs/132689/7

    When I run "snap install core", I still get:

    Code:
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-202714544: unknown filesystem type 'squashfs'.
    
    FYI
    Code:
    # snap install lxd
    error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
           /tmp/sanity-mountpoint-329628364: unknown filesystem type 'squashfs'.
    
    This is what finally has cerbot installed for me.
    Code:
    #apt install certbot
    #apt install python3-certbot-apache
    
    Still trying to figure out how to get it working with ispconfig. Any tips?
     
  14. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Clarity... What I mean is that my status quo is... My primary domain "test.example.com" has a self signed certificate that shows in the browser. However, I am unable to use ispconfig to assignment a SSL certificate to my nextcloud installation. LE.log is working.
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

  16. yupthatguy

    yupthatguy Member HowtoForge Supporter

    My nextcloud installation is installed on my test server under nextcloud.com, but for dev purposes, I use a sitealias tester1.nextcloud.com
    I tried the faq suggestion of "Skip Letsencrypt check" under System -> Server config -> server1.example.com -> Web.

    No Luck. (FYI, I am using a bridged-network adapter)

    EUREKA --- that's the problem... LE is checking acme for my production site (haven't installed nextcloud on production yet)... and clearly acme and certbot aren't friends.

    There is a DNS A record for nextcloud.com on my test server, but LE seems to ignore it in preference of the prod server...
    Code:
    Domain: tester1.nextcloud.com
    Type:   None
    Detail: DNS problem: NXDOMAIN looking up A for tester1.nextcloud.com - check that a DNS record exists for this domain
    2021-06-08 16:31:14,066:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:
    
    Domain: nextcloud.com
    Type:   unauthorized
    Detail: Invalid response from http://nextcloud.com/.well-known/acme-challenge/urhvCLriZJchYlaM4rHsXiDMQz1DsRkkP7tejP1GCsk [prod-ip-xx.xx.xx.xx]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"
    
    To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
    2021-06-08 16:31:14,089:DEBUG:certbot.error_handler:Encountered exception:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
        self._respond(aauthzrs, resp, best_effort)
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
        self._poll_challenges(aauthzrs, chall_update, best_effort)
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
        raise errors.FailedChallenges(all_failed_achalls)
    certbot.errors.FailedChallenges: Failed authorization procedure. tester1.nextcloud.com (http-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for tester1.nextcloud.com - check that a DNS record exists for this domain, nextcloud.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://nextcloud.com/.well-known/acme-challenge/urhvCLriZJchYlaM4rHsXiDMQz1DsRkkP7tejP1GCsk [prod-ip-xx.xx.xx.xx]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"
    
    2021-06-08 16:31:14,089:DEBUG:certbot.error_handler:Calling registered functions
    2021-06-08 16:31:14,089:INFO:certbot.auth_handler:Cleaning up challenges
    2021-06-08 16:31:14,090:DEBUG:certbot.plugins.webroot:Removing /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/urhvCLriZJchYlaM4rHsXiDMQz1DsRkkP7tejP1GCsk
    2021-06-08 16:31:14,090:DEBUG:certbot.plugins.webroot:Removing /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/Re9inhcx2vGCROWKVmzGfJgIljuHAuW0pOEhqt1Dt00
    2021-06-08 16:31:14,090:DEBUG:certbot.plugins.webroot:All challenges cleaned up
    2021-06-08 16:31:14,090:DEBUG:certbot.log:Exiting abnormally:
    Traceback (most recent call last):
      File "/usr/bin/letsencrypt", line 11, in <module>
        load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
        return config.func(config, plugins)
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 1250, in certonly
        lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 121, in _get_and_save_cert
        lineage = le_client.obtain_and_enroll_certificate(domains, certname)
      File "/usr/lib/python3/dist-packages/certbot/client.py", line 410, in obtain_and_enroll_certificate
        cert, chain, key, _ = self.obtain_certificate(domains)
      File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
        orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
      File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
        authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
        self._respond(aauthzrs, resp, best_effort)
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
        self._poll_challenges(aauthzrs, chall_update, best_effort)
      File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
        raise errors.FailedChallenges(all_failed_achalls)
    certbot.errors.FailedChallenges: Failed authorization procedure. tester1.nextcloud.com (http-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for tester1.nextcloud.com - check that a DNS record exists for this domain, nextcloud.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://nextcloud.com/.well-known/acme-challenge/urhvCLriZJchYlaM4rHsXiDMQz1DsRkkP7tejP1GCsk [prod-ip-xx.xx.xx.xx]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"
    2021-06-08 16:31:14,538:DEBUG:certbot.main:certbot version: 0.31.0
    2021-06-08 16:31:14,539:DEBUG:certbot.main:Arguments: ['--domains', 'nextcloud.com', '--domains', 'tester1.nextcloud.com']
    2021-06-08 16:31:14,539:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2021-06-08 16:31:14,545:DEBUG:certbot.log:Root logging level set at 20
    2021-06-08 16:31:14,545:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
    
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Let's encrypt uses a so-called acme challenge to verify domain ownership, so certbot is doing this acme challenge (acme.sh has the program name from this acme challenge, but this does not mean that acme.sh is used on a certbot server). The acme challenge is basically that Let's encrypt tries to reach the domain name from its servers to verify a token, for that reason, you can not get a LE cert for a domain that is not reachable from the internet. It does not matter if you add a local dns record or not for this, Let#s Encrypt will query the official DNS servers and will contact the server IP that these official servers point to for the domain name.
     
  18. yupthatguy

    yupthatguy Member HowtoForge Supporter

    :( I am wrong again... Ok... So how do I generate the self-signed SSL for nextcloud.com on the test server? disconnect the internet, then try?
     
  19. till

    till Super Moderator Staff Member ISPConfig Developer

    On the SSL tab of the website. See ISPConfig manual for step-by-step instructions.
     
  20. yupthatguy

    yupthatguy Member HowtoForge Supporter

    Got it. Thanks.

    Another question...since my prod doesn't have any sites yet, not even nextcloud, is it possible to / how would I go about switching from acme.sh to certbot on my prod server?

    While I don't immediately need ssl on my test server, I know a few wordpress shopping cart plugins require a ssl certificate to be installed, even during dev & testing, so thinking ahead, I don't want the worry of conflicting acme vs. certbot certificates slowing progress...

    How would I adjust these steps to remove acme from my prod server?
     

Share This Page