ISPConfig upgrade 3.1.15 -> 3.2.2 certificate fails

Discussion in 'ISPConfig 3 Priority Support' started by Taleman, Jan 26, 2021.

  1. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Updated Debian 9 to Debian 10 previously. After that the certificates no longer renewed, did not debug much.
    Now acme.sh is installed and upgrading ISPConfig 3.1.15 to 3.2.2 (replaced real domain name with mydomain. It does resolve from name service):
    Code:
    Create new ISPConfig SSL certificate (yes,no) [no]: yes
    Checking / creating certificate for mail.mydomain.fi
    Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
    Using apache for certificate validation
    [ti 26.1.2021 10.00.31 +0200] mail.mydomain.fi:Verify error:Fetching http://mail.mydomain.fi/.well-known/acme-challenge/HoWdtIubVd-_UF1l-43fp7KIzMCS-pfEaRqgt8CkmQQ: Connection refused
    [ti 26.1.2021 10.00.31 +0200] Please add '--debug' or '--log' to check more details.
    [ti 26.1.2021 10.00.31 +0200] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
    Issuing certificate via acme.sh failed. Please check that your hostname can be verified by letsencrypt
    Could not issue letsencrypt certificate, falling back to self-signed.
    genrsa: Can't open "/usr/local/ispconfig/interface/ssl/ispserver.key" for writing, No such file or directory
    Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
    139673506100416:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
    139673506100416:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
    unable to load Private Key
    Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
    140664533984448:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
    140664533984448:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
    unable to load Private Key
    Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
    140154495259840:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
    140154495259840:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
    unable to load Private Key
    PHP Warning:  rename(/usr/local/ispconfig/interface/ssl/ispserver.key.insecure,/usr/local/ispconfig/interface/ssl/ispserver.key): No such file or directory in /tmp/update_stable.sh.m9VrhEVEYE/ispconfig3_install/install/lib/installer_base.lib.php on line 3097
    Reconfigure Crontab? (yes,no) [yes]:
    
    My guess is acme.sh fails getting certificate because apache2 is not running. And apache does not start because certicate error.
    Code:
    [email protected]:~# systemctl --state=failed
      UNIT            LOAD   ACTIVE SUB    DESCRIPTION                   
    ● apache2.service loaded failed failed The Apache HTTP Server       
    ● dovecot.service loaded failed failed Dovecot IMAP/POP3 email server
    
    LOAD   = Reflects whether the unit definition was properly loaded.
    ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
    SUB    = The low-level unit activation state, values depend on unit type.
    
    2 loaded units listed. Pass --all to see loaded but inactive units, too.
    To show all installed unit files use 'systemctl list-unit-files'.
    [email protected]:~# systemctl status apache2.service
    ● apache2.service - The Apache HTTP Server
       Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Tue 2021-01-26 10:04:59 EET; 12min ago
         Docs: https://httpd.apache.org/docs/2.4/
      Process: 13521 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)
    
    tammi 26 10:04:58 mail systemd[1]: Starting The Apache HTTP Server...
    tammi 26 10:04:59 mail apachectl[13521]: AH00548: NameVirtualHost has no effect and will be remove
    tammi 26 10:04:59 mail apachectl[13521]: AH00526: Syntax error on line 188 of /etc/apache2/sites-e
    tammi 26 10:04:59 mail apachectl[13521]: SSLCertificateFile: file '/var/www/clients/client0/web1/s
    tammi 26 10:04:59 mail apachectl[13521]: Action 'start' failed.
    tammi 26 10:04:59 mail apachectl[13521]: The Apache error log may have more information.
    tammi 26 10:04:59 mail systemd[1]: apache2.service: Control process exited, code=exited, status=1/
    tammi 26 10:04:59 mail systemd[1]: apache2.service: Failed with result 'exit-code'.
    tammi 26 10:04:59 mail systemd[1]: Failed to start The Apache HTTP Server.
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    I do
    Code:
    # ls -lh /var/www/clients/client*/web*/ssl
    and see lots of key, crt and bundle files pointing to /etc/letsencrypt/live/. The /etc/letsencrypt is removed already, so all those are missing and Apache complains.
     
  3. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Not much is working, from htf_report.txt:
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Debian GNU/Linux 10 (buster)
     
    [INFO] uptime:  10:45:32 up 9 min,  1 user,  load average: 0,03, 0,11, 0,10
     
    [INFO] memory:
                  total        used        free      shared  buff/cache   available
    Mem:          1,9Gi       1,6Gi       106Mi        26Mi       215Mi       153Mi
    Swap:         1,3Gi        13Mi       1,2Gi
     
    [INFO] ISPConfig is installed.
    
    ##### ISPCONFIG #####
    ISPConfig version is 3.2.2
    
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 7.3.26-1+0~20210112.74+debian10~1.gbpd78724
    [INFO] php-cgi (used for cgi php in default vhost!) is version 5.6.40-0+deb8u2
    
    ##### PORT CHECK #####
    
    [WARN] Port 8080 (ISPConfig) seems NOT to be listening
    [WARN] Port 8081 (ISPConfig Apps) seems NOT to be listening
    [WARN] Port 80 (Webserver) seems NOT to be listening
    [WARN] Port 443 (Webserver SSL) seems NOT to be listening
    [WARN] Port 143 (IMAP server) seems NOT to be listening
    [WARN] Port 993 (IMAP server SSL) seems NOT to be listening
    [WARN] Port 110 (POP3 server) seems NOT to be listening
    [WARN] Port 995 (POP3 server SSL) seems NOT to be listening
    
    ##### MAIL SERVER CHECK #####
    
    
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you use certbot before? In this case, I won't install acme.sh as all old certs will fail then. Also take care to never delete /etc/letsencrypt on an active server, as the webserver will stop working then and also the ispconfig update must fail to issue a cert as the web server config is already completely damaged due to missing /etc/letsencrypt folder. Either restore /etc/letsencrypt from a backup to get your server in a working state again or do this:

    If you don't have a backup, then you must disable all sites that use ssl in /etc/apache2/sites-available folder, bit not the ispconfig and apps vhost. then restart apache, login to ispconfig and activate ssl and let's encrypt for each site one by one to get new ssl certs.
     
    Th0m likes this.
  5. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    It seems (from your second post) that you deleted /etc/letsencrypt. You should not have done that. Apache cannot run because each vhost has missing certificates now and so you are not able to issue new ones.

    What you need to do now is:
    • either restore /etc/letsencrypt with all contents, so apache can start again
    • or delete all vhost files from /etc/apache2/sites-enabled that are using non-existent LE cert files, then remove all the invalid symlinks from the /*/ssl/ paths, then restart apache and re-save each website from the ISPConfig UI to re-generate the certificates and vhost files.
     
  6. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Should I uninstall acme.sh and apt-get install certbot? Then reconfigure services with ispconfig?
    Websites work but for some reason logging in to ISPConfig panel as admin fails.
     
  7. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Depends.
    Have you restored /etc/letsencrypt? Then uninstall acme and install certbot (while the apt-get version is quite outdated …).
    If you have not restored /etc/letsencrypt then stick with acme and re-create the certs.
     
  8. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    I restored /etc/letsencrypt from backup, and websites work. Still can not figure out why I can not login to ISPConfig panel. So I can not recreate certificates ...
    Debian 10 has certbot 0.31.0-1 but backports buster has python3-certbot-dns-gandi (1.2.5-2~bpo10+1).
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Which error do you get in the browser and which error in the global web server error.log?
     
  10. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Now logging to panel works. Go figure.
    No luck today, still problems. Running ispconfig_update.sh --force. Got complaints instance of certbot was already running, so I did systemctl stop certbot.service and tried again:

    Code:
    Create new ISPConfig SSL certificate (yes,no) [no]: yes
    
    Checking / creating certificate for mail.mydomain.fi
    Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
    Using apache for certificate validation
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Cert is due for renewal, auto-renewing...
    Renewing an existing certificate
    Performing the following challenges:
    http-01 challenge for mail.mydomain.fi
    Using the webroot path /usr/local/ispconfig/interface/acme for all unmatched domains.
    Waiting for verification...
    Cleaning up challenges
    Failed authorization procedure. mail.mydomain.fi (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.mydomain.fi/.well-known/acme-challenge/9duCjmYR1rEcnZC5LNISCT0ibwoXeDhj55-ZH8AjTHw: Connection refused
    Issuing certificate via certbot failed. Please check log files and make sure that your hostname can be verified by letsencrypt
    Could not issue letsencrypt certificate, falling back to self-signed.
    Generating RSA private key, 4096 bit long modulus (2 primes)
    
     
  11. Croydon

    Croydon ISPConfig Developer ISPConfig Developer

    Code:
    grep -r '9duCjmYR1rEcnZC5LNISCT0ib' /var/log/apache2 /var/log/ispconfig/httpd
     
  12. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    That command finds nothing.
    My interpretation is that apache went down again. So apache is not running and LE fails to verify.
    And again, apache2 fails to start.
     
  13. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    I can not figure this. Apache is not running:
    Code:
    [email protected]:/var/log/apache2# systemctl --state=failed
      UNIT            LOAD   ACTIVE SUB    DESCRIPTION                                     
    ● apache2.service loaded failed failed The Apache HTTP Server                         
    ● certbot.service loaded failed failed Certbot                                         
    ● dovecot.socket  loaded failed failed Dovecot IMAP/POP3 email server activation socket
    
    
    And it can not start
    Code:
    Create new ISPConfig SSL certificate (yes,no) [no]: yes
    
    Checking / creating certificate for mail.mydomain.fi
    Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
    Using apache for certificate validation
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Cert is due for renewal, auto-renewing...
    Renewing an existing certificate
    Performing the following challenges:
    http-01 challenge for mail.mydomain.fi
    Using the webroot path /usr/local/ispconfig/interface/acme for all unmatched domains.
    Waiting for verification...
    Cleaning up challenges
    Failed authorization procedure. mail.mydomain.fi (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.mydomain.fi/.well-known/acme-challenge/9duCjmYR1rEcnZC5LNISCT0ibwoXeDhj55-ZH8AjTHw: Connection refused
    Issuing certificate via certbot failed. Please check log files and make sure that your hostname can be verified by letsencrypt
    Could not issue letsencrypt certificate, falling back to self-signed.
    Generating RSA private key, 4096 bit long modulus (2 primes)
    
    ispconfig_update.sh can not issue LE certificate because apache is not running.
    I could restore this host from snapshot, but I do not understand why it messed certificatet this morning so this may happen again.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    Check if the ssl cert and key in /usr/local/ispconfig/interface/ssl/ are there and that they point to a valid ssl cert.
     
  15. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    I do not understand why apache fails to start. This from apache error log:
    Code:
    [ 2021-01-26 16:31:32.3224 721/7f79e52e1980 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished
    [ 2021-01-26 16:36:29.0587 5448/7f2f9ea3d980 age/Wat/WatchdogMain.cpp:1291 ]: Starting Passenger watchdog...
    [ 2021-01-26 16:36:29.0814 5451/7f8273f63980 age/Cor/CoreMain.cpp:982 ]: Starting Passenger core...
    [ 2021-01-26 16:36:29.0815 5451/7f8273f63980 age/Cor/CoreMain.cpp:235 ]: Passenger core running in multi-application mode.
    [ 2021-01-26 16:36:29.0833 5451/7f8273f63980 age/Cor/CoreMain.cpp:732 ]: Passenger core online, PID 5451
    [ 2021-01-26 16:36:29.0928 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
    [ 2021-01-26 16:36:29.0945 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 5458
    AH00016: Configuration Failed
    
    [ 2021-01-26 16:36:29.1204 5451/7f8273674700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
    [ 2021-01-26 16:36:29.1205 5451/7f8273f63980 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
    
    [ 2021-01-26 16:36:29.1206 5458/7fb68ac90700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
    [ 2021-01-26 16:36:29.1206 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
    [ 2021-01-26 16:36:29.1206 5458/7fb68a40e700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
    [ 2021-01-26 16:36:29.1207 5458/7fb68a40e700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
    [ 2021-01-26 16:36:29.1208 5458/7fb68ac90700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
    [ 2021-01-26 16:36:29.1209 5451/7f8273674700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects
    [ 2021-01-26 16:36:29.1209 5451/7f8273674700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
    [ 2021-01-26 16:36:29.1210 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
    [ 2021-01-26 16:36:29.1218 5451/7f8272df2700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
    [ 2021-01-26 16:36:29.1219 5451/7f8272df2700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
    
    [ 2021-01-26 16:36:29.1253 5451/7f8273674700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 1 more time(s) to force shutdown)
    [ 2021-01-26 16:36:29.1460 5451/7f8273f63980 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished
    
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    When apache does not throw an error, then the most likely reason is a missing SSL cert, as apache will not show any errors in most cases if that happens, it fails just silently. The main problem is that it#s not easy to detect as you'll basically have to check all ssl certs. But as this issue happened after you run an update, the most likely reason is that it's the ssl cert from the ispconfig vhost, and that#s why I suggested to check that.
     
  17. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    In /usr/local/ispconfig/interface/ssl the ispserver.key is ordinary file, not a symlink. I'll try making it symlink to the corresponding entry in letsencrypt files.
    Copying the set of files with same datestamp from backup worked, apache starts. Now to figure out how I break this setup and not do it again, and check sites do have a valid certificate.
     
  18. budgierless

    budgierless Member HowtoForge Supporter

    I have this same problem, how did you fix this?
    i just updated from 3.1.15p3 to 3.2.4 few moments ago, now have got i have same problem, im still using ubuntu 18.04, but all sites are working but no access to control panel.
    Code:
    Can't open /usr/local/ispconfig/interface/ssl/ispserver.crt for writing, No such file or directory
    140485853753792:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('/usr/local/ispconfig/interface/ssl/ispserver.crt','w')
    140485853753792:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
    writing RSA key
    cat: /usr/local/ispconfig/interface/ssl/ispserver.crt: No such file or directory
    Symlink ISPConfig SSL certs to Postfix? (y,n) [y]:
     
  19. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I think you have missing required lib files or something, so retrace / redo all required steps in ISPConfig PST normally fix such error(s) and, if that are all done, try run ispconfig force update again.

    Before that I think it is better that you to check and remove old method of securing ispconfig panel (by undoing all of the steps) as it will be conflicting with ISPConfig 3.2++, if you followed that steps before.
     
  20. budgierless

    budgierless Member HowtoForge Supporter

    I did forget to mention that I used your uninstall script prior to the update, so just to confirm I already did this step that you suggest, I can try to run the script again but this was not mentioned previously in this thread as a Cause

    letsencrypt.log
    Code:
    2021-05-14 12:17:34,249:DEBUG:certbot.main:certbot version: 0.27.0
    2021-05-14 12:17:34,250:DEBUG:certbot.main:Arguments: ['--agree-tos', '--non-interactive', '--expand', '--rsa-key-size', '4096', '--server', 'https://acme-v02.api.letsencrypt.org/directory', '--authenticator', 'webroot', '--webroot-path', '/usr/local/ispconfig/interface/acme', '--email', '[email protected]', '-d', 'server1.example.com', '--renew-hook', 'letsencrypt_renew_hook.sh']
    2021-05-14 12:17:34,251:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2021-05-14 12:17:34,284:DEBUG:certbot.log:Root logging level set at 20
    2021-05-14 12:17:34,285:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
    2021-05-14 12:17:34,286:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
    2021-05-14 12:17:34,286:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
    Description: Place files in webroot directory
    Interfaces: IAuthenticator, IPlugin
    Entry point: webroot = certbot.plugins.webroot:Authenticator
    Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f30f35b4710>
    Prep: True
    2021-05-14 12:17:34,288:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f30f35b4710> and installer None
    2021-05-14 12:17:34,288:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
    2021-05-14 12:17:34,302:DEBUG:certbot.log:Exiting abnormally:
    Traceback (most recent call last):
      File "/usr/bin/letsencrypt", line 11, in <module>
        load_entry_point('certbot==0.27.0', 'console_scripts', 'certbot')()
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
        return config.func(config, plugins)
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 1238, in certonly
        le_client = _init_le_client(config, auth, installer)
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 641, in _init_le_client
        acc, acme = _determine_account(config)
      File "/usr/lib/python3/dist-packages/certbot/main.py", line 512, in _determine_account
        acc = display_ops.choose_account(accounts)
      File "/usr/lib/python3/dist-packages/certbot/display/ops.py", line 83, in choose_account
        "Please choose an account", labels, force_interactive=True)
      File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 512, in menu
        self._interaction_fail(message, cli_flag, "Choices: " + repr(choices))
      File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 474, in _interaction_fail
        raise errors.MissingCommandlineFlag(msg)
    certbot.errors.MissingCommandlineFlag: Missing command line flag or config entry for this setting:
    Please choose an account
    Choices: ['[email protected]:19:22Z (8d53)', '[email protected]:46:07Z (6193)']
     
    Last edited: May 14, 2021

Share This Page