Apache 2 fails to run overnight - Certificate and private key do not match

Discussion in 'ISPConfig 3 Priority Support' started by BartelBe, Jul 19, 2021 at 12:42 PM.

  1. BartelBe

    BartelBe New Member HowtoForge Supporter

    Running ISPConfig 3.2 on Debian 10 smoothly for months, my Apache2 webserver decided to stop working and fails to start.
    Obviously it is essential to have Apache2 running.
    The logs show only Certificate and private key xxx:8080:0 from /usr/local/ispconfig/interface/ssl/ispserver.crt and /usr/local/ispconfig/interface/ssl/ispserver.key do not match
    (edit) Checking the acme log, it appears that the SSL for domain I use to run ispconfig failed to renew:
    vibel.be:Verify error:Invalid response from http://vibel.be/.well-known/acme-challenge (which is not on the system)

    Any help on getting my apacheserver back online would be greatfully accepted.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    So the server uses a hostname that points to another system? Does the command:

    hostname

    or

    hostname -f

    returns this 'wrong' hostname?
     
  3. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    What do you mean with:
    Try the following:
    Open /etc/apache2/sites-available/ispconfig.vhost
    Comment this out:
    Code:
    # SSL Configuration
      SSLEngine On
        SSLProtocol All -SSLv3 -TLSv1 -TLSv1.1
        SSLCertificateFile /usr/local/ispconfig/interface/ssl/ispserver.crt
      SSLCertificateKeyFile /usr/local/ispconfig/interface/ssl/ispserver.key
      #SSLCACertificateFile /usr/local/ispconfig/interface/ssl/ispserver.bundle
    
      SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECD$
      SSLHonorCipherOrder On
    Start apache:
    Code:
    systemctl start apache2
    Run a force update to retrieve a new cert. Share the output of the update script with us:
    Code:
    ispconfig_update.sh --force
     
    till likes this.
  4. BartelBe

    BartelBe New Member HowtoForge Supporter

    No not at all.
    The correct hostname is returned
     
  5. BartelBe

    BartelBe New Member HowtoForge Supporter


    Thank you.
    After ALSO disabling these in apps.vhost, I got Apache2 running again. However, I cannot connect to ISPConfig (on port 8080).
    Certificate and private key nlhb52619ds.vibel.be:8081:0 from /usr/local/ispconfig/interface/ssl/ispserver.crt and /usr/local/ispconfig/interface/ssl/ispserver.key do not match
    (edit) As a result of this new error I went looking and found the port 8081 in apps.vhost and I commented out the SSL configuration there too.

    And re: "what do you mean with" http://vibel.be/.well-known/acme-challenge <-- this path does not exist, it is not present. The acme log gave this as reason for the failed response of vibel.be

    (proceeding with iispconfig_update.sh --force)
     
    Last edited: Jul 19, 2021 at 2:15 PM
  6. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  7. till

    till Super Moderator Staff Member ISPConfig Developer

    That#s a virtual path, not real path in filesystem. So it shall not exist in the website. This virtual path is mapped to /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/
     
  8. BartelBe

    BartelBe New Member HowtoForge Supporter

    Using apache for certificate validation
    acme.sh is installed, overriding certificate path to use /root/.acme.sh/nlhb52619ds.vibel.be
    [Mon 19 Jul 2021 02:56:25 PM CEST] nlhb52619ds.vibel.be:Verify error:Invalid response from http://nlhb52619ds.vibel.be/.well-known/acme-challenge/GV8aJMsr_yBZYGPFTrxlPlP6mPJLO54B_T-5lbktIbM [2606:4700:3036::6815:1120]:
    [Mon 19 Jul 2021 02:56:25 PM CEST] Please check log file for more details: /var/log/ispconfig/acme.log
    Issuing certificate via acme.sh failed. Please check that your hostname can be verified by letsencrypt
    Could not issue letsencrypt certificate, falling back to self-signed.
    Generating a RSA private key
    ...........................................................................................................++++
    .++++
    writing new private key to '/usr/local/ispconfig/interface/ssl/ispserver.key'

    Letsencrypt searches for /.well-known/acme-challenge, which path does not exist ... AND issueingvia acme.sh failed....
    This makes accessing ispconfig impossible...
     
  9. BartelBe

    BartelBe New Member HowtoForge Supporter

  10. till

    till Super Moderator Staff Member ISPConfig Developer

    That's not what the message above says. It says that a token which it created in the folder /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/ is not reachable from outside of your server by http://.

    You can test it like this. Run the command:

    touch /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/test.txt

    you must now be able to access this file with this URL to receive a valid LE SSL cert:

    http://nlhb52619ds.vibel.be/.well-known/acme-challenge/test.txt

    if this fails, then check if DNS really points to the right server (IPv4 and IPv6) and that you don't block port 80 in a firewall.
     
  11. BartelBe

    BartelBe New Member HowtoForge Supporter

    You were right assuming that "if this fails, then check if DNS really points to the right server". Cloudflare DNS proxied the IP address. I had to correct that to enable it to directly connect to the IP address and checked that it does.
    However running [ispconfig_update.sh --force] again after verifying that the test file is reacheable and readable, failed again
    [Mon 19 Jul 2021 03:46:51 PM CEST] nlhb52619ds.vibel.be:Verify error:Fetching http://nlhb52619ds.vibel.be/.well-known/acme-challenge/H97oxzhqtgX3i5hFEFcVgnJETHZZzYOtAnEoag78PnI: Connection refused
    [Mon 19 Jul 2021 03:46:51 PM CEST] Please check log file for more details: /var/log/ispconfig/acme.log
    Issuing certificate via acme.sh failed. Please check that your hostname can be verified by letsencrypt
    Could not issue letsencrypt certificate, falling back to self-signed.
    Generating a RSA private key
     
    Last edited: Jul 19, 2021 at 3:48 PM
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you comment out the SSL config lines again like @Th0m suggested before running the forced upgrade again? If no, please try the full procedure again by commenting out these lines, then doing a forced update.
     
  13. BartelBe

    BartelBe New Member HowtoForge Supporter

    I redid the commenting out of the SSL config lines on ispconfig.vhost AND the apps.vhost, then restarted apache2 and then sudo ispconfig_update.sh --force --> same result

    Checking / creating certificate for nlhb52619ds.vibel.be
    Using certificate path /root/.acme.sh/nlhb52619ds.vibel.be
    Using apache for certificate validation
    acme.sh is installed, overriding certificate path to use /root/.acme.sh/nlhb52619ds.vibel.be
    [Mon 19 Jul 2021 04:21:18 PM CEST] nlhb52619ds.vibel.be:Verify error:Fetching http://nlhb52619ds.vibel.be/.well-known/acme-challenge/YqMGoOgMSeSTc_UZzkm9qK6mJSuWnfAlelT2QNe1ukY: Connection refused

    But at least, apache2 is up again and the websites are live. Only access to the ISPConfig interface is prohibited (or accessible through forced https access exception)

    Here is the ispconfig acme log
    [Mon 19 Jul 2021 05:51:37 PM CEST] Running cmd: issue
    [Mon 19 Jul 2021 05:51:37 PM CEST] _main_domain='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:37 PM CEST] _alt_domains='no'
    [Mon 19 Jul 2021 05:51:37 PM CEST] Using config home:/root/.acme.sh
    [Mon 19 Jul 2021 05:51:37 PM CEST] default_acme_server='https://acme-v02.api.letsencrypt.org/directory'
    [Mon 19 Jul 2021 05:51:37 PM CEST] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
    [Mon 19 Jul 2021 05:51:37 PM CEST] DOMAIN_PATH='/root/.acme.sh/nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:37 PM CEST] Using ACME_DIRECTORY: https://acme-v02.api.letsencrypt.org/directory
    [Mon 19 Jul 2021 05:51:37 PM CEST] _init api for server: https://acme-v02.api.letsencrypt.org/directory
    [Mon 19 Jul 2021 05:51:37 PM CEST] GET
    [Mon 19 Jul 2021 05:51:37 PM CEST] url='https://acme-v02.api.letsencrypt.org/directory'
    [Mon 19 Jul 2021 05:51:37 PM CEST] timeout=
    [Mon 19 Jul 2021 05:51:37 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:37 PM CEST] ret='0'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_KEY_CHANGE='https://acme-v02.api.letsencrypt.org/acme/key-change'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_NEW_AUTHZ
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_NEW_ORDER='https://acme-v02.api.letsencrypt.org/acme/new-order'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_NEW_ACCOUNT='https://acme-v02.api.letsencrypt.org/acme/new-acct'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_REVOKE_CERT='https://acme-v02.api.letsencrypt.org/acme/revoke-cert'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_AGREEMENT='https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'
    [Mon 19 Jul 2021 05:51:38 PM CEST] ACME_NEW_NONCE='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
    [Mon 19 Jul 2021 05:51:38 PM CEST] Le_NextRenewTime
    [Mon 19 Jul 2021 05:51:38 PM CEST] Using CA: https://acme-v02.api.letsencrypt.org/directory
    [Mon 19 Jul 2021 05:51:38 PM CEST] _on_before_issue
    [Mon 19 Jul 2021 05:51:38 PM CEST] _chk_main_domain='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:38 PM CEST] _chk_alt_domains
    [Mon 19 Jul 2021 05:51:38 PM CEST] Le_LocalAddress
    [Mon 19 Jul 2021 05:51:38 PM CEST] d='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:38 PM CEST] Check for domain='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:38 PM CEST] _currentRoot='/usr/local/ispconfig/interface/acme'
    [Mon 19 Jul 2021 05:51:38 PM CEST] d
    [Mon 19 Jul 2021 05:51:38 PM CEST] _saved_account_key_hash is not changed, skip register account.
    [Mon 19 Jul 2021 05:51:38 PM CEST] Read key length:
    [Mon 19 Jul 2021 05:51:38 PM CEST] _createcsr
    [Mon 19 Jul 2021 05:51:38 PM CEST] Single domain='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:38 PM CEST] Getting domain auth token for each domain
    [Mon 19 Jul 2021 05:51:38 PM CEST] d
    [Mon 19 Jul 2021 05:51:38 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/new-order'
    [Mon 19 Jul 2021 05:51:38 PM CEST] payload='{"identifiers": [{"type":"dns","value":"nlhb52619ds.vibel.be"}]}'
    [Mon 19 Jul 2021 05:51:38 PM CEST] RSA key
    [Mon 19 Jul 2021 05:51:38 PM CEST] HEAD
    [Mon 19 Jul 2021 05:51:38 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
    [Mon 19 Jul 2021 05:51:38 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g -I '
    [Mon 19 Jul 2021 05:51:39 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:39 PM CEST] POST
    [Mon 19 Jul 2021 05:51:39 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/new-order'
    [Mon 19 Jul 2021 05:51:39 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:40 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:40 PM CEST] code='201'
    [Mon 19 Jul 2021 05:51:40 PM CEST] Le_LinkOrder='https://acme-v02.api.letsencrypt.org/acme/order/124272533/11192376274'
    [Mon 19 Jul 2021 05:51:40 PM CEST] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/124272533/11192376274'
    [Mon 19 Jul 2021 05:51:40 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/14959992950'
    [Mon 19 Jul 2021 05:51:40 PM CEST] payload
    [Mon 19 Jul 2021 05:51:40 PM CEST] POST
    [Mon 19 Jul 2021 05:51:40 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/authz-v3/14959992950'
    [Mon 19 Jul 2021 05:51:40 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:40 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:40 PM CEST] code='200'
    [Mon 19 Jul 2021 05:51:40 PM CEST] d='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:40 PM CEST] Getting webroot for domain='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:40 PM CEST] _w='/usr/local/ispconfig/interface/acme'
    [Mon 19 Jul 2021 05:51:40 PM CEST] _currentRoot='/usr/local/ispconfig/interface/acme'
    [Mon 19 Jul 2021 05:51:40 PM CEST] entry='"type":"http-01","status":"pending","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA","token":"Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U"'
    [Mon 19 Jul 2021 05:51:40 PM CEST] token='Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U'
    [Mon 19 Jul 2021 05:51:40 PM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:40 PM CEST] keyauthorization='Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U.0STKIhSzQpWD-NJb2ZuSv9JIfEi0ASkA21dZnpiDHtk'
    [Mon 19 Jul 2021 05:51:40 PM CEST] dvlist='nlhb52619ds.vibel.be#Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U.0STKIhSzQpWD-NJb2ZuSv9JIfEi0ASkA21dZnpiDHtk#https://acme-v02.api.letsencrypt.or...A#http-01#/usr/local/ispconfig/interface/acme'
    [Mon 19 Jul 2021 05:51:40 PM CEST] d
    [Mon 19 Jul 2021 05:51:40 PM CEST] vlist='nlhb52619ds.vibel.be#Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U.0STKIhSzQpWD-NJb2ZuSv9JIfEi0ASkA21dZnpiDHtk#https://acme-v02.api.letsencrypt.or...A#http-01#/usr/local/ispconfig/interface/acme,'
    [Mon 19 Jul 2021 05:51:41 PM CEST] d='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:41 PM CEST] ok, let's start to verify
    [Mon 19 Jul 2021 05:51:41 PM CEST] Verifying: nlhb52619ds.vibel.be
    [Mon 19 Jul 2021 05:51:41 PM CEST] d='nlhb52619ds.vibel.be'
    [Mon 19 Jul 2021 05:51:41 PM CEST] keyauthorization='Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U.0STKIhSzQpWD-NJb2ZuSv9JIfEi0ASkA21dZnpiDHtk'
    [Mon 19 Jul 2021 05:51:41 PM CEST] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:41 PM CEST] _currentRoot='/usr/local/ispconfig/interface/acme'
    [Mon 19 Jul 2021 05:51:41 PM CEST] wellknown_path='/usr/local/ispconfig/interface/acme/.well-known/acme-challenge'
    [Mon 19 Jul 2021 05:51:41 PM CEST] writing token:Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U to /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U
    [Mon 19 Jul 2021 05:51:41 PM CEST] Changing owner/group of .well-known to ispconfig:ispconfig
    [Mon 19 Jul 2021 05:51:41 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:41 PM CEST] payload='{}'
    [Mon 19 Jul 2021 05:51:41 PM CEST] POST
    [Mon 19 Jul 2021 05:51:41 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:41 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:41 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:41 PM CEST] code='200'
    [Mon 19 Jul 2021 05:51:41 PM CEST] trigger validation code: 200
    [Mon 19 Jul 2021 05:51:41 PM CEST] sleep 2 secs to verify
    [Mon 19 Jul 2021 05:51:43 PM CEST] checking
    [Mon 19 Jul 2021 05:51:43 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:43 PM CEST] payload
    [Mon 19 Jul 2021 05:51:43 PM CEST] POST
    [Mon 19 Jul 2021 05:51:43 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:43 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:44 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:44 PM CEST] code='200'
    [Mon 19 Jul 2021 05:51:44 PM CEST] nlhb52619ds.vibel.be:Verify error:Fetching http://nlhb52619ds.vibel.be/.well-known/acme-challenge/Ctj-jgnb4-xBbhUMxdn_xANhovMbSO9Zk-z50YijG2U: Connection refused
    [Mon 19 Jul 2021 05:51:44 PM CEST] pid
    [Mon 19 Jul 2021 05:51:44 PM CEST] No need to restore nginx, skip.
    [Mon 19 Jul 2021 05:51:44 PM CEST] _clearupdns
    [Mon 19 Jul 2021 05:51:44 PM CEST] dns_entries
    [Mon 19 Jul 2021 05:51:44 PM CEST] skip dns.
    [Mon 19 Jul 2021 05:51:44 PM CEST] _on_issue_err
    [Mon 19 Jul 2021 05:51:44 PM CEST] Please check log file for more details: /var/log/ispconfig/acme.log
    [Mon 19 Jul 2021 05:51:44 PM CEST] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:44 PM CEST] payload='{}'
    [Mon 19 Jul 2021 05:51:44 PM CEST] POST
    [Mon 19 Jul 2021 05:51:44 PM CEST] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/14959992950/bUTEmA'
    [Mon 19 Jul 2021 05:51:44 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:51:45 PM CEST] _ret='0'
    [Mon 19 Jul 2021 05:51:45 PM CEST] code='400'
    [Mon 19 Jul 2021 05:52:44 PM CEST] Running cmd: upgrade
    [Mon 19 Jul 2021 05:52:44 PM CEST] Using config home:/root/.acme.sh
    [Mon 19 Jul 2021 05:52:44 PM CEST] default_acme_server='https://acme-v02.api.letsencrypt.org/directory'
    [Mon 19 Jul 2021 05:52:44 PM CEST] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
    [Mon 19 Jul 2021 05:52:44 PM CEST] GET
    [Mon 19 Jul 2021 05:52:44 PM CEST] url='https://api.github.com/repos/acmesh-official/acme.sh/git/refs/heads/master'
    [Mon 19 Jul 2021 05:52:44 PM CEST] timeout=
    [Mon 19 Jul 2021 05:52:44 PM CEST] _CURL='curl --silent --dump-header /root/.acme.sh/http.header -L -g '
    [Mon 19 Jul 2021 05:52:44 PM CEST] ret='0'
    [Mon 19 Jul 2021 05:52:44 PM CEST] Already uptodate!
    [Mon 19 Jul 2021 05:52:44 PM CEST] Upgrade success!
    [Mon 19 Jul 2021 05:52:44 PM CEST] Running cmd: setdefaultca
    [Mon 19 Jul 2021 05:52:44 PM CEST] Changed default CA to: https://acme-v02.api.letsencrypt.org/directory
     
    Last edited: Jul 19, 2021 at 6:04 PM

Share This Page