Inconsistent SSL configurations

Discussion in 'ISPConfig 3 Priority Support' started by Jemt, Jul 30, 2020.

  1. Jemt

    Jemt Member HowtoForge Supporter


    I have two websites: X and Y

    On X, file_get_contents(..) works fine for posting to the website itself via https.
    On Y, file_get_contents(..) throws the following error when posting to the website itself via https:

    PHP Warning: file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages:
    mod_fcgid: stderr: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in FILENAME
    mod_fcgid: stderr: PHP Warning: file_get_contents(): Failed to enable crypto in FILENAME
    mod_fcgid: stderr: PHP Warning: file_get_contents(https://YYYYYYYY):$
    mod_fcgid: stderr: PHP Fatal error: Uncaught Exception: Request to URL 'https://YYYYYYYYY

    The one significant difference I can find is the two websites' certificates.

    X's vhost file contains:
    SSLCertificateFile /var/www/clients/client1/web4/ssl/XXX-le.crt
    SSLCertificateKeyFile /var/www/clients/client1/web4/ssl/XXX-le.key

    Y's vhost file contains:
    SSLCertificateFile /var/www/clients/client10/web75/ssl/YYY-le.crt
    SSLCertificateKeyFile /var/www/clients/client10/web75/ssl/YYY-le.key

    Looks identical. However, what's interesting is the symlinks to these files.

    Content of X's certificate folder (notice how XXX-le.crt points to fullchain.pem):
    [email protected]:/# ls -la /var/www/clients/client1/web4/ssl/
    total 68
    drwxr-xr-x 2 root root 4096 Jan 18 2019 .
    drwxr-xr-x 11 root root 4096 Dec 4 2017 ..
    lrwxrwxrwx 1 root root 44 Jan 18 2019 XXX-le.bundle -> /etc/letsencrypt/live/XXX/chain.pem
    -r-------- 1 root root 1647 Dec 4 2017 XXX-le.bundle.old.20171204194502
    -r-------- 1 root root 1647 Dec 4 2017 XXX-le.bundle.old.20171204194902
    -r-------- 1 root root 1647 Feb 16 2018 XXX-le.bundle.old.20180216075913
    -r-------- 1 root root 1647 Feb 16 2018 XXX-le.bundle.old.20180216080003
    -r-------- 1 root root 1647 Jan 18 2019 XXX-le.bundle.old.20190118191504
    lrwxrwxrwx 1 root root 48 Jan 18 2019 XXX-le.crt -> /etc/letsencrypt/live/XXX/fullchain.pem
    -r-------- 1 root root 2159 Dec 4 2017 XXX-le.crt.old.20171204194502
    -r-------- 1 root root 3806 Dec 4 2017 XXX-le.crt.old.20171204194902
    -r-------- 1 root root 3830 Feb 16 2018 XXX-le.crt.old.20180216075913
    -r-------- 1 root root 3830 Feb 16 2018 XXX-le.crt.old.20180216080003
    -r-------- 1 root root 3944 Jan 18 2019 XXX-le.crt.old.20190118191504
    lrwxrwxrwx 1 root root 46 Jan 18 2019 XXX-le.key -> /etc/letsencrypt/live/XXX/privkey.pem
    -r-------- 1 root root 3272 Dec 4 2017 XXX-le.key.old.20171204194502
    -r-------- 1 root root 3272 Dec 4 2017 XXX-le.key.old.20171204194902
    -r-------- 1 root root 3272 Feb 16 2018 XXX-le.key.old.20180216075913
    -r-------- 1 root root 3272 Feb 16 2018 XXX-le.key.old.20180216080003
    -r-------- 1 root root 3272 Jan 18 2019 XXX-le.key.old.20190118191504

    Content of Y's certificate folder (notice how YYY-le-crt points to cert.pem)
    [email protected]:/# ls -la /var/www/clients/client10/web75/ssl/
    total 12
    drwxr-xr-x 2 root root 4096 Sep 18 2017 .
    drwxr-xr-x 11 root root 4096 Jul 29 22:24 ..
    lrwxrwxrwx 1 root root 58 Sep 18 2017 YYY-le.bundle -> /etc/letsencrypt/live/YYY/chain.pem
    lrwxrwxrwx 1 root root 57 Sep 18 2017 YYY-le.crt -> /etc/letsencrypt/live/YYY/cert.pem
    lrwxrwxrwx 1 root root 60 Sep 18 2017 YYY-le.key -> /etc/letsencrypt/live/YYY/privkey.pem

    I know close to little about SSL, but my understand is that fullchain.pem is the file contains both the certificate and intermediate certificates while cert.pem only contains the certificate - and we need the fullchain.pem file for file_get_contents(..) to work properly.

    1) Why are the two websites different ?
    2) Can I fix the Y website (and potentially other broken websites) without modifying the vhost files by hand or changing symlinks? This is a production server, so I can't risk breaking things - I must be able to upgrade packages and ISPConfig. Also, I'd like both old and future website to behave the same.

    - Thanks in advance

    Best Regards
    Jimmy Thomsen
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    If /etc/letsencrypt/live/YYY/cert.pem does not contain the chain certs as well, copy them from /etc/letsencrypt/live/YYY/chain.pem file and add them to the end of the file. Might be that it was created with a old certbot version that did not create fullchain certs or something similar.
  3. Jemt

    Jemt Member HowtoForge Supporter

    @till, thanks for your suggestion. I just created a new Let's Encrypt certificate for a new website, and that too does not have the full chain in the cert.pem file. But that's how it's supposed to work (

    [email protected]:/# ls -la /etc/letsencrypt/live/ZZZZZZZ/
    total 12
    drwxr-xr-x 2 root root 4096 Aug 1 12:43 .
    drwx------ 13 root root 4096 Aug 1 12:42 ..
    lrwxrwxrwx 1 root root 31 Aug 1 12:42 cert.pem -> ../../archive/ZZZZZZZ/cert1.pem
    lrwxrwxrwx 1 root root 32 Aug 1 12:42 chain.pem -> ../../archive/ZZZZZZZ/chain1.pem
    lrwxrwxrwx 1 root root 36 Aug 1 12:42 fullchain.pem -> ../../archive/ZZZZZZZ/fullchain1.pem
    lrwxrwxrwx 1 root root 34 Aug 1 12:42 privkey.pem -> ../../archive/ZZZZZZZ/privkey1.pem
    -rw-r--r-- 1 root root 692 Aug 1 12:42 README
    [email protected]:/#

    1) Would it make sense to have ISPConfig reference fullchain.pem in vhosts instead I wonder?
    I'm not familiar with the potential security implications.
    2) Rather than modifying the files, wouldn't it make better sense to change the symlink like so?:
    /var/www/clients/client10/web75/ssl/YYY-le.crt => /etc/letsencrypt/live/YYY/cert.pem
    /var/www/clients/client10/web75/ssl/YYY-le.crt => /etc/letsencrypt/live/YYY/fullchain.pem
    This is what the working website is doing and it seems to have been working for a while.
    Are you familiar with anything that would break ? Will ISPConfig overwrite /var/www/clients/client10/web75/ssl/YYY-le.crt
    and have it point to the cert.pem file again at some point, or will ISPConfig only "ensure" these symlinks if they are not already there ?

    - Thanks

  4. Jemt

    Jemt Member HowtoForge Supporter

    @till, or would it perhaps be possible to point to a specific certificate file using the Apache Directives under Options for the website? I'm not sure how it works. If I specify
    SSLCertificateFile /var/www/clients/client0/web109/ssl/custom-symlink-to-fullchain.crt
    it will simply add that line at the bottom of both <VirtualHost *:80> and <VirtualHost *:443>, and crash Apache.
    Can I somehow make it merge into <VirtualHost *:443> and have it replace the existing SSLCertificateFile directive added by ISPConfig?

    I'm trying to figure out which solution would be the most reliable as I continue to upgrade the OS and ISPConfig.
  5. Jemt

    Jemt Member HowtoForge Supporter

    Sorry @till, I only just now realized that the new website with a new certificate has its certificate symlink in the ssl directory pointing to fullchain.pem and not chain.pem. So that's good.

    I tried fixing the problem with my Y website by renaming YYY-le.crt (symlink to /etc/letsencrypt/live/YYY/cert.pem) to __YYY-le.crt and created a new symlink:
    YYY-le.crt -> /etc/letsencrypt/live/YYY/fullchain.pem

    However, that caused SSLLabs' SSL checker ( to report
    "This server's certificate chain is incomplete. Grade capped to B."

    I finally got fed up and removed the SSL certificate (unchecked it under the website settings), gave it a couple of minutes, and checked it again, and everything now works fine. The symlink now points to fullchain.pem, and the SSL checker reports no errors. Guess I should have tried that hours ago :)


Share This Page