[SOLVED] pure-ftpd using old certificate

Discussion in 'Installation/Configuration' started by Thibaut, May 12, 2020.

  1. Thibaut

    Thibaut New Member

    I've searched the forum and found similar issues but couldn't find an answer to the problem I'm facing.

    Here is the situation:
    I have an ISPConfig 3.1.15p2 installed on Debian 10, strictly following the steps described in https://www.howtoforge.com/perfect-server-debian-10-buster-apache-bind-dovecot-ispconfig-3-1/
    I've been able to FTP to the server using SSL/TLS before, but now, when I try to connect today, FileZilla reports an "Unknown Certifiate Error"
    The connection log being as follow:
    Status:   Resolving address of my.server.tld
    Status:   Connecting to X.X.X.X:21...
    Status:   Connection established, waiting for welcome message...
    Status:   Initializing TLS...
    Status:   Verifying certificate...
    Error:   Remote certificate not trusted.
    Error:   Critical error: Could not connect to server
    Status:   Disconnected from server
    Trying to us lftp also fails:
    $ lftp
    lftp :~> set ftp:ssl-force true
    lftp :~> connect my.server.tld
    lftp my.serve.tld:~> login user_name
    lftp [email protected]:~> ls
    ls: Fatal error: Certificate verification: Not trusted (07:AE:09:86:35:90:D3:11:7A:B0:4D:3D:82:49:8B:52:CB:44:F5:A8)
    lftp [email protected]:~>
    The pure-ftpd server was configured to use the same Let's Encrypt certificate that is used for the main ISPConfig interface (https://my.server.tld:8080). Which means I have a combined .pem file under /usr/local/ispconfig/interface/ssl/ispserver.pem, pure-ftp was configured to use TLS (echo 1 > /etc/pure-ftpd/conf/TLS) and a symlink to the ispserver.pem file was created under /etc/ssl/private/pure-ftpd.pem.

    The Let's Encrypt autorenewal script is present under /etc/init.d/le_ispc_pem.sh, containing the following code:
    # Required-Start: $local_fs $network
    # Required-Stop: $local_fs
    # Default-Start: 2 3 4 5
    # Default-Stop: 0 1 6
    # Short-Description: LE ISPSERVER.PEM AUTO UPDATER
    # Description: Update ispserver.pem automatically after ISPC LE SSL certs are renewed.
    cd /usr/local/ispconfig/interface/ssl/
    mv ispserver.pem ispserver.pem-$(date +"%y%m%d%H%M%S").bak
    cat ispserver.{key,crt} > ispserver.pem
    chmod 600 ispserver.pem
    chmod 600 /etc/ssl/private/pure-ftpd.pem
    systemctl restart pure-ftpd-mysql
    systemctl restart monit
    systemctl restart postfix
    systemctl restart dovecot
    systemctl restart apache2
    And it was activated using incrontab -e
    /etc/letsencrypt/archive/my.server.tld/    IN_MODIFY       ./etc/init.d/le_ispc_pem.sh
    All of this is working as expected since it is possible to access the ISPConfig web interface (https://my.server.tld:8080) and display the certificate which currently shows (today is 05/12/2020):
    Subject Name
    Common Name my.server.tld
    Issuer Name
    Country US
    Organization Let's Encrypt
    Common Name Let's Encrypt Authority X3
    Not Before 4/6/2020, 4:05:45 AM (Central European Summer Time)
    Not After 7/5/2020, 4:05:45 AM (Central European Summer Time)
    Subject Alt Names
    DNS Name my.server.tld
    Thus, no problem using https.

    It is possible to see that autorenewal script has worked as expected since listing /usr/local/ispconfig/interface/ssl/ outputs:
    ls -la /usr/local/ispconfig/interface/ssl/
    lrwxrwxrwx 1 root      root        54 Dec  7 16:02 ispserver.crt -> /etc/letsencrypt/live/my.server.tld/fullchain.pem
    lrwxrwxrwx 1 root      root        52 Dec  7 16:02 ispserver.key -> /etc/letsencrypt/live/my.server.tld/privkey.pem
    -rw------- 1 root      root      7184 Apr  6 03:05 ispserver.pem
    -rw-r--r-- 1 root      root      7183 Feb  6 03:05 ispserver.pem-200206030524.bak
    -rw-r--r-- 1 root      root      7184 Apr  6 03:05 ispserver.pem-200406030546.bak
    And the pure-ftp symlink is correct:
    ls -la /etc/ssl/private/
    -rw-r--r-- 1 root root      424 Nov 25 05:51 pure-ftpd-dhparams.pem
    lrwxrwxrwx 1 root root       48 Dec  7 16:20 pure-ftpd.pem -> /usr/local/ispconfig/interface/ssl/ispserver.pem
    -rw-r----- 1 root ssl-cert 1704 Jul  8  2019 ssl-cert-snakeoil.key
    I've tried receating the /etc/ssl/private/pure-ftpd.pem symlink, and of course restarting pure-ftpd using
    systemctl restart pure-ftpd-mysql
    but the situation didn't change.

    It really seems like pure-ftpd is still using the previous certificate (which was valid until 05/06/2020), but I have no clue how to figure out what to do to remedy this situation.
    Any help would be welcome.

    Thank you very much.
  2. ahrasis

    ahrasis Well-Known Member

    Did you chmod 600 after recreating /etc/ssl/private/pure-ftpd.pem symlink before restarting pure-ftpd?
  3. Thibaut

    Thibaut New Member

    Thank you @ahrasis for paying attention to my problem.
    Yes, I did chmod 600 /etc/ssl/private/pure-ftpd.pem, although I noticed that this operation doesn't change the symlink's rights on Debian. I also tried chmod -h 600, but this returned an error:
    chmod: invalid option -- 'h'         
    Try 'chmod --help' for more information.
    Which led me to the following: https://www.gnu.org/software/coreutils/manual/html_node/chmod-invocation.html#chmod-invocation where it is stated that:
    Apparently, chmod -h would work on FreeBSD/openBSD/MacOS but not on Linux systems.

    Nevertheless, the configuration initially used to work, even though the symlink itself never got the right permissions, since setting the target file rights as -rw------- seems all we can do on Debian (Linux) systems.

    I'm still stuck with the problem :(
    Last edited: May 12, 2020
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Just to be sure, did you check that the pen file contains the right cert? try:

    cat /etc/ssl/private/pure-ftpd.pem

    the result must be the content of
  5. Thibaut

    Thibaut New Member

    Hello @till, and thank you for taking the time to have a look at my problem.
    Your suggestion was indeed relevant, and using diff on the cert files pointed out that the current content of /etc/ssl/private/pure-ftpd.pem was not up to date!

    Since there are several symlinks involved, I started by "drawing" the symlink chains in order to get a clearer view of the situation:
    1]  /etc/ssl/private/pure-ftpd.pem  -->  /usr/local/ispconfig/interface/ssl/ispserver.pem
    2]  /usr/local/ispconfig/interface/ssl/ispserver.key  -->  /etc/letsencrypt/live/my.server.tld/privkey.pem  -->  /etc/letsencrypt/archive/my.server.tld/privkey3.pem
    3]  /usr/local/ispconfig/interface/ssl/ispserver.crt  -->  /etc/letsencrypt/live/my.server.tld/fullchain.pem -->  /etc/letsencrypt/archive/my.server.tld/fullchain3.pem
    I first verified 2] and 3] which are OK.
    Then I verified that 1] == (2] + 3]) and that was NOT OK.

    As can be seen from 2] and 3], there have been 2 Let's Encrypt certificates updates since the system was installed, making it so that the content of /etc/letsencrypt/archive/my.server.tld/ currently is:
    > ls -la /etc/letsencrypt/archive/my.server.tld/
    total 56
    drwxr-xr-x  2 root root 4096 Apr  6 03:05 .
    drwx------ 14 root root 4096 Mar  5 11:10 ..
    -rw-r--r--  1 root root 2264 Dec  7 15:46 cert1.pem
    -rw-r--r--  1 root root 2269 Feb  6 03:05 cert2.pem
    -rw-r--r--  1 root root 2264 Apr  6 03:05 cert3.pem
    -rw-r--r--  1 root root 1647 Dec  7 15:46 chain1.pem
    -rw-r--r--  1 root root 1647 Feb  6 03:05 chain2.pem
    -rw-r--r--  1 root root 1647 Apr  6 03:05 chain3.pem
    -rw-r--r--  1 root root 3911 Dec  7 15:46 fullchain1.pem
    -rw-r--r--  1 root root 3916 Feb  6 03:05 fullchain2.pem
    -rw-r--r--  1 root root 3911 Apr  6 03:05 fullchain3.pem
    -rw-------  1 root root 3272 Dec  7 15:46 privkey1.pem
    -rw-------  1 root root 3268 Feb  6 03:05 privkey2.pem
    -rw-------  1 root root 3268 Apr  6 03:05 privkey3.pem
    Indicating that the first certificate was issued on Dec 7 (2019), then renewed on Feb 6 (2020) and again on Apr 6 (2020). Which is coherent with the content of /usr/local/ispconfig/interface/ssl/ where we see:
    > ls -la /usr/local/ispconfig/interface/ssl/                                                                 
    total 52
    drwxr-x--- 2 root      root      4096 Apr  6 03:05 .
    drwxr-x--- 9 ispconfig ispconfig 4096 Nov 25 07:14 ..
    lrwxrwxrwx 1 root      root        54 Dec  7 16:02 ispserver.crt -> /etc/letsencrypt/live/my.server.tld/fullchain.pem
    lrwxrwxrwx 1 root      root        52 Dec  7 16:02 ispserver.key -> /etc/letsencrypt/live/my.server.tld/privkey.pem
    -rw------- 1 root      root      7184 Apr  6 03:05 ispserver.pem
    -rw-r--r-- 1 root      root      7183 Feb  6 03:05 ispserver.pem-200206030524.bak
    -rw-r--r-- 1 root      root      7184 Apr  6 03:05 ispserver.pem-200406030546.bak
    The presence of both .bak files clearly indicates that the /etc/init.d/le_ispc_pem.sh script has run twice. Now the incoherence I surprisingly observed is that the content of ispserver.pem, which we can see was last touched at the time the privkey3.pem and fullchain3.pem files were created, does contain privkey2.pem and fullchain2.pem, when it should contain privkey3.pem and fullchain3.pem!

    I then manually ran the /etc/init.d/le_ispc_pem.sh script which behaved exactly as expected, creating a new /usr/local/ispconfig/interface/ssl/ispserver.pem-200512224542.bak file and, this time, copying privkey3.pem and fullchain3.pem in the /usr/local/ispconfig/interface/ssl/ispserver.pem file.
    My opinion is that, maybe, incron triggers the update script too soon when the Let's Encrypt renewal script touches the /etc/letsencrypt/archive/my.server.tld/ directory, even before the /etc/letsencrypt/live/my.server.tld/privkey.pem --> /etc/letsencrypt/archive/my.server.tld/privkeyX.pem (and others) symlinks get updated?

    Do you think this might be the cause?
    Anyway, since I'm pretty sure it won't brake anything, I have added the following as the first line of executable code in the /etc/init.d/le_ispc_pem.sh script:
    # Description: Update ispserver.pem automatically after ISPC LE SSL certs are renewed.
    sleep 2s
    cd /usr/local/ispconfig/interface/ssl/
    mv ispserver.pem ispserver.pem-$(date +"%y%m%d%H%M%S").bak
    cat ispserver.{key,crt} > ispserver.pem
    I'm also pretty sure that 2s are overkill but I don't see any reason not to be safe here.
    PS: to be honest I also added this line at the end of the script in order to remind me to check if everything is OK after the next certificate update ;)
    mail -s 'Lets Encrypt updated my.server.tld' [email protected] -a From:[email protected] <<< 'Check incron correctly created the /usr/local/ispconfig/interface/ssl/ispserver.pem file.'
    I'd be interrested in your opinion regarding the cause of the problem I've encountered, and I'll certainly report here after the next certificate update to let you know wether it happened again or not.
    In the meantime, thanks again to you guys @ahrasis and @till, who took the time to answer me.
    Last edited: May 13, 2020
    till and ahrasis like this.
  6. ahrasis

    ahrasis Well-Known Member

    Thank you for the sleep suggestion. Agree incron could have triggered almost immediately then acted a little bit faster before LE renewal process is completed, hence the problem.
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    I agree with you that that the problem was probably a race condition where incron executed the cat command too early, so the 2 sec sleep is a good idea.
  8. Thibaut

    Thibaut New Member

    As promised, just a quick feedback to confirm that the FTPD certificate was well updated this time!
    Marking this thread as SOLVED ;)
    Thanks again for the input.
    ahrasis likes this.

Share This Page