Wrong Cert Served

Discussion in 'Installation/Configuration' started by Gray Consulting, Aug 26, 2020.

  1. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Hello:

    We host an ISP Config 3.1 server, Ubuntu 20.04, PHP 7.4, Apache2, MySQL

    Currently we host 9 websites, seven of which have valid SSL certificates installed. (All Comodo, we don't use LetsEncrypt or other self-signed certificates). Of those 7 sites, 2 are suffering some SSL failure, so a 'default' cert is served up instead, causing a domain/cert name mismatch error

    Code:
    NET::ERR_CERT_COMMON_NAME_INVALID
    This server could not prove that it is domain2.com; its security certificate is from domain1.com.
    We're logging SSL activity (LogLevel debug), but the only entries we get are from Apache restarts.

    Code:
    [Wed Aug 26 16:12:08.948478 2020] [ssl:info] [pid 76774] AH01914: Configuring server domain1.com:443 for SSL protocol
    [Wed Aug 26 16:12:08.948614 2020] [ssl:debug] [pid 76774] ssl_engine_init.c(497): AH01893: Configuring TLS extension handling
    [Wed Aug 26 16:12:08.948622 2020] [ssl:debug] [pid 76774] ssl_util_stapling.c(929): AH01960: OCSP stapling initialized
    [Wed Aug 26 16:12:08.948806 2020] [ssl:debug] [pid 76774] ssl_util_ssl.c(444): AH02412: [domain1.com:443] Cert matches for name 'domain1.com' [subject: CN=domain1.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 3649C77B154FF8B40987190B91D79D4B / notbefore: Aug 26 00:00:00 2020 GMT / notafter: Sep 27 00:00:00 2021 GMT]
    [Wed Aug 26 16:12:08.948815 2020] [ssl:info] [pid 76774] AH02568: Certificate and private key domain1.com:443:0 configured from /var/www/clients/client1/web11/ssl/domain1.com.crt and /var/www/clients/client1/web11/ssl/domain1.com.key
    [Wed Aug 26 16:12:08.948915 2020] [ssl:error] [pid 76774] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=domain1.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 3649C77B154FF8B40987190B91D79D4B / notbefore: Aug 26 00:00:00 2020 GMT / notafter: Sep 27 00:00:00 2021 GMT]
    [Wed Aug 26 16:12:08.948922 2020] [ssl:error] [pid 76774] AH02604: Unable to configure certificate domain1.com:443:0 for stapling
    At first we thought we were onto the problem, noting the failure to staple the certificate, and the missing CA certificate. But the other domain suffering the same symptoms, shows stapling was successful, and was fine with the exact same CA certificate

    Code:
    [Wed Aug 26 16:12:08.948927 2020] [ssl:info] [pid 76774] AH01914: Configuring server domain2.com:443 for SSL protocol
    [Wed Aug 26 16:12:08.949058 2020] [ssl:debug] [pid 76774] ssl_engine_init.c(497): AH01893: Configuring TLS extension handling
    [Wed Aug 26 16:12:08.949065 2020] [ssl:debug] [pid 76774] ssl_util_stapling.c(929): AH01960: OCSP stapling initialized
    [Wed Aug 26 16:12:08.949331 2020] [ssl:debug] [pid 76774] ssl_util_ssl.c(444): AH02412: [domain2.com:443] Cert matches for name 'domain2.com' [subject: CN=domain2.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 8D15AAF1386A8702558C18627FE365CF / notbefore: Aug 20 00:00:00 2020 GMT / notafter: Sep 23 23:59:59 2020 GMT]
    [Wed Aug 26 16:12:08.949339 2020] [ssl:info] [pid 76774] AH02568: Certificate and private key domain2.com:443:0 configured from /var/www/clients/client1/web5/ssl/domain2.com.crt and /var/www/clients/client1/web5/ssl/domain2.com.key
    The other domains where SSL is working show identical successful certificate loading with each restart of Apache. Could not find any related errors in /var/log/apache2/error.log or log/error.log. (Is there a way to turn on debug level for the apache log?)

    Happy to share real domain names if need be, I know that's otherwise avoided

    Boiled down, mod_ssl is fine with the certificate when initializing at Apache startup, but https requests are failing.

    Everything in the ispconfig configs for the various sites seem correct, and with no differences that we could identify.

    All the cert directives in the associated .vhost files point to the correct path / file names; we've even double-checked the keys and certificates on the server, against what was actually issued. Everything checks.

    Happy to provide any config detail, error logs, or the like. As always, any help or direction would be greatly appreciated.

    htf_report here:
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Ubuntu 20.04.1 LTS
    
    [INFO] ISPConfig is installed.
    [WARN] /usr/local/ispconfig/server/lib/config.inc.php is missing.
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 7.4.3
    
    ##### PORT CHECK #####
    
    ##### MAIL SERVER CHECK #####
    
    ##### RUNNING SERVER PROCESSES #####
    
    [WARN] I could not determine which web server is running.
    [WARN] I could not determine which mail server is running.
    [WARN] I could not determine which pop3 server is running.
    [WARN] I could not determine which imap server is running.
    [WARN] I could not determine which ftp server is running.
    
    ##### LISTENING PORTS #####
    (only        ()
    Local        (Address)
    [anywhere]:21        (-)
    ***.***.***.***:53        (-)
    [localhost]:53        (-)
    ***.***.***.***:53        (-)
    [anywhere]:22        (-)
    [anywhere]:25        (-)
    [localhost]:953        (-)
    [anywhere]:993        (-)
    [anywhere]:995        (-)
    [localhost]:10023        (-)
    [localhost]:10024        (-)
    [localhost]:10025        (-)
    [localhost]:10026        (-)
    [localhost]:10027        (-)
    [anywhere]:587        (-)
    [localhost]:11211        (-)
    [anywhere]:110        (-)
    [anywhere]:143        (-)
    [anywhere]:465        (-)
    *:*:*:*::*:21        (-)
    *:*:*:*::*a6:e8ff:feb7:1:53        (-)
    *:*:*:*::*:53        (-)
    *:*:*:*::*:22        (-)
    *:*:*:*::*:25        (-)
    *:*:*:*::*:953        (-)
    *:*:*:*::*:443        (-)
    *:*:*:*::*:993        (-)
    *:*:*:*::*:995        (-)
    *:*:*:*::*:10024        (-)
    *:*:*:*::*:10026        (-)
    *:*:*:*::*:3306        (-)
    *:*:*:*::*:587        (-)
    [localhost]10        (-)
    [localhost]43        (-)
    *:*:*:*::*:8080        (-)
    *:*:*:*::*:80        (-)
    *:*:*:*::*:8081        (-)
    *:*:*:*::*:465        (-)
    
    ##### IPTABLES #####
     
    Last edited: Aug 27, 2020
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Take care that all websites use * or all websites use the IP address, do not mix * and IP on a server in the IPv4 field of the websites.
     
  3. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Thanks Till -
    All good there; every site uses '*'.
    Are there any other logs I could investigate? None of the php or apache logs have anything, nor do the 'custom' debug logs we configured for the various domains in the ssl.c section of the .vhost file. I thought there would be a way to turn on debug-level logging for any ssl events, but not seeing it.

    (update; found how to turn on the debug firehose in Apache2; now seeing some related errors. Will update this post with results... )
    Thanks!
     
    Last edited: Aug 27, 2020
  4. Jesse Norell

    Jesse Norell ISPConfig Developer ISPConfig Developer

    If you accept the "bad" certificate, the access logs should show your request, at least. Does that happen?
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    And it will help to know which content you see, the content of the correct site or from another site.
     
  6. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Thanks guys -
    The access logs are tricky to interpret, as they only shows internal IP's, e.g.:
    Code:
    172.31.13.170 - - [27/Aug/2020:16:16:27 +0000] "-" 408 0 "-" "-"
    The http error 408 seems odd; it's certainly not timeout behavior we're seeing. And if I'm reading it right, the request header is empty, which would match what I found in the apache error log:
    Code:
    [Thu Aug 27 16:43:29.123075 2020] [reqtimeout:info] [pid 150686] [client 172.31.16.89:40104] AH01382: Request header read timeout
    ... and what should be the matching entry from the access log:
    Code:
    172.31.16.89 - - [27/Aug/2020:16:43:29 +0000] "-" 408 0 "-" "-"
    If the header really is empty, where might I look to fix that?

    Thanks as always ~
     
  7. Steini86

    Steini86 Active Member

    If you run "apache2ctl -S", is there an entry for the domain in question for both, port 80 and 443?
     
  8. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Yes - for the site in question (all sites, in fact), we see entries for both 80 and 443
     
  9. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Gentlemen:

    No success as yet, but considering whacking a whole website record in ISPConfig, and starting with a fresh one. Not sure it will solve anything, but after everything else I've tried, figured it might help. I have excellent site and ISPConfig backups, so all good there. Any red flags that you see in this plan?

    Thanks as always ~
     
  10. Th0m

    Th0m ISPConfig Developer ISPConfig Developer

    I wouldn't do that, because it would take quite some time and I usually avoid using backups when it's not necessary.

    You can try disabling SSL, re-enabling it, and eventually remove and then add the certificate again for the sites that are 'redirecting'
     
  11. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Thanks Thom -
    I have tried toggling SSL off/on, with no luck, but I will try removing & reinstalling the certs themselves. (I'm still convinced that it's something with the certs themselves, or their config, but can't prove it with any logs...)
    Cheers ~
     
  12. Th0m

    Th0m ISPConfig Developer ISPConfig Developer

    If that doesn't work, post/PM me your domain names, so I can check
     
  13. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Thanks Thom, but unlucky - after deleting & reinstalling cert, no change. I did notice that after installing the cert, I looked on the server, and confirmed the crt file contains both certs (site cert and CA cert); I assume that's correct

    intenttopay.com - logs show apache/ssl happy with the right cert, when restarting apache, but redirects
    umsturz.net - logs show apache/ssl fails stapling cert, but otherwise seems to 'load' cert ok, but redirects
    gcjaguar.net - fyi only; this is the default cert being served. We don't actually host a site under this domain; we us it as our server 'root'. E.g., postfix mail server is ns1.gcjaguar.net; we manage admin email under webmail.gcjaguar.net

    Not ruling out desperate measures ;)
     
  14. Th0m

    Th0m ISPConfig Developer ISPConfig Developer

    I did a DNS lookup for the A records of your domain names. The first 2 have 2 A records, pointing to the same servers. The third points to a completely different IP address. Could it be that your traffic goes to a different server?
    upload_2020-9-1_0-21-49.png
     
    Last edited: Sep 1, 2020
    Gray Consulting likes this.
  15. Gray Consulting

    Gray Consulting Member HowtoForge Supporter

    Et, voila!

    Thom, you nailed it - we had mis-configured our load-balancer, and once fixed, both sites are validating ssl/tls. I couldn't see the forest for the trees...

    A huge thanks again to you and Till - do you guys have a tip jar? ;)
     
  16. Th0m

    Th0m ISPConfig Developer ISPConfig Developer

    Glad to hear :) A reinstallation would have been a waste of time.

    For the tip jar, I'll leave that question to @till
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    We don't :) But maybe we add one in future.
     
    Th0m likes this.

Share This Page