letsencrypt on mail server

Discussion in 'Tips/Tricks/Mods' started by Jesse Norell, Aug 9, 2016.

  1. zenny

    zenny Member

    Hi, I manually got the certificates from LE server with several mailserver domains (in my case, certbot certonly --standalone -d mail.domain1.tld -d mail.domain2.tld -d mail.domain3.tld -d mail.domain4.cld and so on) Tried with four domains and DNS records are pointed to the same IP. Both /etc/postfix/smtpd.{key, crt} were linked to the relevant files in the /etc/letsencrypt/live/mail.domain1.tld/ directory. All the services were rebooted (postfix, dovecot and nginx), but the letsencrypt server did not seem to came into effect (still getting the privately signed certificate). Where did I miss? Platform is Debian8 Jessie with ISPConfig 3.1.2. Thanks!
     
  2. Jesse Norell

    Jesse Norell Well-Known Member

    Verify the letsencrypt certificate got created; what does this show?:
    Code:
    ls -l /etc/letsencrypt/live/mail.domain1.tld/
    ls -lH /etc/letsencrypt/live/mail.domain1.tld/*
    openssl x509 -in /etc/letsencrypt/live/mail.domain1.tld/cert.pem -noout -subject -issuer
    
    Assuming the certificates are created, my next guess is the services wen't actually restarted, but just "reloaded". Try a full stop/start. If that still doesn't get it, you have to look at them service by service.

    For postfix: make sure you are using /etc/postfix/smtpd.cert (not .crt). If that's correct, the postfix smtpd.{key,cert} symlinks didn't get created, or postfix is simply not pointing to them. What do these commands show:
    Code:
    ls -l /etc/postfix/smtpd.{key,cert,crt}
    ls -lH /etc/postfix/smtpd.{key,cert,crt}
    postconf smtpd_tls_CAfile smtpd_tls_cert_file smtpd_tls_key_file
    
    For dovecot: assuming the previous symlinks are in order, make sure dovecot is using them:
    Code:
    doveconf ssl_key ssl_cert
    
    nginx wasn't covered here, but is in other places. like above, make sure the server points to the right place for the cert/key; check both that the webserver config point somewhere, and that that somewhere symlinks to the correct place.
     
    zenny likes this.
  3. Jesse Norell

    Jesse Norell Well-Known Member

    I just updated the initial post with a cronjob which compares the certificates postfix and dovecot are using with the latest from letsencrypt and restarts services if they differ. This is important for stable systems that don't restart for months at a time, so the certificate in memory will update.
     
    liane likes this.
  4. liane

    liane Member HowtoForge Supporter

    this script is really nice, but perhaps you should display it plain, because I guess you meant it to be run as is from the terminal and install the cron and the script in one pass, but alas, bash gets in the way and intercepts backtickts (like `${OPENSSL} x509 -noout -serial -in ${LE_CERT}`) instead of writing them to the file
     
  5. Jesse Norell

    Jesse Norell Well-Known Member

    Sorry, yes, that probably doesn't work right (the cat <<EOF > file stuff was all added for convenience, and untested). I'll update again.
     
  6. Jesse Norell

    Jesse Norell Well-Known Member

    The initial post is updated again, and I tweaked the monitoring script some so it is actually untested at this point, too - holler if anyone has problems.
     
  7. kerrsmith

    kerrsmith Member

    After reading the thread (https://www.howtoforge.com/community/threads/on-renewing-letsencrypt.75837/) and it being stated that ISPConfig would renew all certificates even if they were manually generated I decided to look in to why my server did not do this - below are my findings (this may help others and also me if I forget what I did):

    First I had a look at /var/log/letsencrypt/letsencrypt.log and saw the following:

    Description: Spin up a temporary webserver
    Entry point: standalone = certbot.plugins.standalone:Authenticator
    DEBUG:certbot.error_handler:Encountered exception:
    MisconfigurationError: At least one of the required ports is already taken.

    It appeared it was trying to renew but as I had used standalone when creating the certificate this is what it was trying to do again for the renewal process but as I have Apache running this process was failing.

    I had a read up on alternative methods and saw instead of --standalone I could use --webroot. As all unknown domains that hit the server get shown the default web page (in ISPConfig's case the first one of your hosted domains alphabetically) I had a look at using this (I have a site www.domain.com but not pop3.domain.com). I tried unsuccessfully at first getting this working as the .well-known/acme-challenge folder/files appeared in the default web root but this was not accessible via the default website even if I manually created it as a test.

    I searched a bit more and found the following file:

    /etc/apache2/sites-enabled/000-ispconfig.conf

    This file contains an alias:

    Alias /.well-known/acme-challenge /usr/local/ispconfig/interface/acme/.well-known/acme-challenge

    So I now did the following:

    ./certbot-auto certonly --text --agree-tos --webroot --email [email protected] --webroot-path /usr/local/ispconfig/interface/acme/ -d domain.com -d pop3.domain.com -d smtp.domain.com

    This now successfully created the certificates. This morning I checked the log and there are no longer any errors so it looks like the certificates should now get renewed by ISPConfig automatically.
     
    Last edited: Mar 16, 2017
    till likes this.
  8. Jesse Norell

    Jesse Norell Well-Known Member

    Thanks for the note, @kerrsmith, I'll update the initial instructions. You do need to use standalone mode for a dedicated mail server, but indeed it will fail if it can't answer on port 80.
     
  9. webguyz

    webguyz Active Member HowtoForge Supporter

    Jesse,
    Have you ever installed LE on an email cluster? both the master and the slave will have the same domain name but wondering if LE will freak with there being 2 IP's when it does a DNS validation check
    A mymail.acme.com 192.168.0.5
    A mymail.acme.com 192.168.0.6
    Thanks!
     
  10. Jesse Norell

    Jesse Norell Well-Known Member

    No, I haven't. If the hostname has 2 ip addrs, I would suspect (but haven't confirmed) that they will randomly pick one to connect to for the validation check, so both of them would need to be able to answer. You could probably redirect all requests to one server (eg. via port forwards in a network firewall maybe, or on right on the second server), or setup a real webserver on each and file-share the well-known challenge directory (so both servers can access the temp file that is created there).

    Another option would be use DNS-based validation for those hostnames, eg. by using acme.sh.
     
  11. sjau

    sjau Local Meanie Moderator

  12. Ark74

    Ark74 Member

    Hi!
    By using:
    hostname -f
    then should I include all mail.domain.ltxyz domains on /etc/hostname ?
    As for right now, I only have the hostname for the server. AFAIK this is not part of the Perfect Tutorial series.
     
  13. Jesse Norell

    Jesse Norell Well-Known Member

    Include whatever names you want included in the certificate. It'd be great to include all customer domains (within the limit of 100 per certificate), but I don't do that - it just seems too prone to problems. Eg. a customer transfers their domain and/or email away, or lets their domain expire, and doesn't tell you, so there's now a "bad" hostname in the certificate - in <90 days your server can't renew it's certificate and all your other customers get errors because the certificate has expired.
     
  14. Airbag888

    Airbag888 New Member

    @Jesse Norell @till Sorry if I'm necroing. I went through https://www.howtoforge.com/tutorial/securing-ispconfig-3-with-a-free-lets-encrypt-ssl-certificate/ and although at some point you point the simlink in /etc/postfix to the /etc/letsencrypt/live files the mail server (residing on the web server btw as mail.domain.com) is not pushing the correct certificate.

    So I'm curious, did I do a step wrong or is it still correct that your tutorial should be executed after doing the above to have a mail server that is connected to via ssl. Btw will it be SSL or TLS for pop3/imap/smtp ?

    Thanks!
     
  15. Jesse Norell

    Jesse Norell Well-Known Member

    This thread and the other one you linked to both setup your mail server with a certificate, but differ in the commands, file location, etc. Use one or the other, trying to mix them could get you a bad setup.

    Both, ssl on port 465, 993 and 995, and tls on 25, 110, 143 and 587.
     
  16. Airbag888

    Airbag888 New Member

    Many thanks for this quick answer. Please bear with nub me.

    My setup is domain.com and mx pointing to mail.domain.com running the latest ispconfig (3.1.16)

    Managed to letsencrypt properly after a few failures (snapshots save the day - dns propagation sucks)

    So following the first link I only created a certificate for domain.com and attaching it in postfix/dovecot (via symlink) warned users of the fact that this was not a cert for mail.domain.com

    So I ran the cert tool to generate a cert for mail.domain.com (had to stop apache2), symlinked that one instead et voila! Done.

    Now couple of new issues:
    1. I don't seem to be able to connect to port 993/995, only 110 (using TLS)
    2. What happens when I add a new client/domain (domainB.com) ? I could keep on using mail.domain.com and have the right cert OR ideally use mail.domainB.com.

    Two fold question, a) creating this in ispconfig and checking SSL + Let's Encrypt certificate checkbox will create a www certificate?
    b) Will it also do the same for the mail or will I need to manually create another certificate for mail.domainB.com ?

    I've read somewhere postfix is SNI and not SNA or something like that (no idea what it means) but it points to a limit of 100 domains (plenty for me btw) ?


    Finally I've been having some odd issues in ISPconfig (fresh install on ubuntu 18 LTS following the perfect tutorial/guide)
    1. Creating a DNS zone with wizard does nothing at all (nothing in apache2 error logs)
    2. https://domain.com worked good but http://domain.com (1. did not auto redirect to https, 2. pointed to the default site so I'd see a ubuntu welcome apache page

    Once /etc/apache2/sites-enabled/100-fantasticmind.io.vshost was edited as below

    <VirtualHost *:80> -> <VirtualHost 111.222.333.444:80>
    111.222.333.444 being the server's IP of course.

    The correct site was linked...

    I wonder if that is linked to the letsencrypt process or not...

    Thanks!!
     
  17. Jesse Norell

    Jesse Norell Well-Known Member

    You can (should?) create one certificate with multiple names in it, eg. domain.com and mail.domain.com.

    Maybe restart dovecot and look for errors in the mail log. I don't see any explicit config to enable those ports, I think (guess?) they are just done by default when you have a certificate specified.

    You could add mail.domainB.com to the certificate, or just have the new client use mail.domain.com as their server name. Read this whole thread and particularly note comment 33: https://www.howtoforge.com/community/threads/letsencrypt-on-mail-server.73695/page-2#post-363901

    That depends how you created the certificate in the first place; this thread does not use letsencrypt via the ISPConfig interface to do that, the other thread you linked to does.

    More likely it's related to the ip addresses you use for your websites (use all '*' or all ip addresses, but don't mix the two).
     
  18. Chris_UK

    Chris_UK Member HowtoForge Supporter

    Hope this helps somebody as I missed A step (not on ths thread but in general).

    Working on Ubuntu 16.04 standalone mail server with UFW
    Log into ssh as root

    if you have not installed certbot already
    Code:
    add-apt-repository ppa:certbot/certbot
    apt update
    apt install certbot
    
    From here you can used LE4ISPC script in the pinned post of this forum or use the manual steps as above,

    One thing to note, for me port 443 alone was not enough, the http_01 authentication step also required opening port 80. so this is what I had to do:
    Code:
    ufw allow 80 && ufw allow 443
    ufw reload && ufw status
    
    Here you now get the cert from Jessie directions above or run the automated script, don't forget you can use the dry run flag in the certbot command by adding --dry-run

    Once you have your cert, close the ports again (if you want to, no server is accepting requests on these ports)

    Code:
    ufw status numbered
    # Find the number of the rule you want to remove and adjust the next command as needed
    # execute the two commands as often as needed until all rules you want to remove are gone
    ufw delete 1
    # finally reload ufw
    ufw reload
    
    Thats it, your server should be working off its cert and opened ports closed again, ofcourse you need to look at what to do to open the ports and close them again if you want automatic renewals.
     
    Last edited: Mar 23, 2019
  19. vincentkersten

    vincentkersten New Member

    Another shot at the mailserver certs-to-install - in this script i am pulling the names from the ispconfig database. Hope this is useful to someone else too.

    Code:
    #!/bin/bash
    # quickie to get all the mailnames in a cert for ISPCONFIG
    # NOTE: all mail domain names, including the ispserver mail domain (mail.my.ispserver.com) need to have a DNS entry or it will fail. 
    
    die () {
    	echo "ERROR: $1"
    	exit 1
    }
    
    
    # Is certbot installed?
    which certbot-auto &>/dev/null || die "Certbot-auto not found, please install it first"
    
    # See that we do not overload the cert
    count=0
    
    # get the domain of the host
    ISPDOMAIN=$(hostname -d)
    ISPSERVER=$(hostname -f)
    
    #build the certbot line with on-line auth
    CERTLINE="certbot-auto auth --text --agree-tos --webroot --webroot-path /usr/local/ispconfig/interface/acme/ --email [email protected]$ISPDOMAIN -d $ISPSERVER -d mail.$ISPSERVER"
    
    # use this for standalone (not tested)
    #CERTLINE="certbot-auto auth --text --agree-tos --standalone --email [email protected]$ISPDOMAIN -d $ISPSERVER -d mail.$ISPSERVER"
    
    # get the names from the ispconfig database on this host
    # spit out all domains to ALLDOMAINS, separated by space
    # watch out for backticks and EOF; this is overly sensitive
    ALLDOMAINS=`/usr/bin/env php << 'EOF'
    <?php
    require "/usr/local/ispconfig/server/lib/config.inc.php";
    require "/usr/local/ispconfig/server/lib/app.inc.php";
    if ($app->dbmaster->connect_error == NULL) {
    $server_db_record = $app->dbmaster->queryAllRecords("SELECT domain FROM mail_domain");
    if(!is_array($server_db_record)) die('Unable to load the server configuration from database.');
    foreach( $server_db_record as $key => $val) {
    echo $val['domain']." ";
    } } ; ?> 
    EOF`
    
    # complete the certbot line by putting 'mail.' in front of all available domains
    for DOMAIN in $ALLDOMAINS ; do  
    	CERTLINE="$CERTLINE -d $DOMAIN -d mail.$DOMAIN"
    	let count++
    done
    
    # we can add 100 domains
    [ $count -gt 49 ] && die "Too much ($count) domains found for including in one cert in $0"
    
    #request the cert
    $CERTLINE || die "Failed to get the certficate with: $CERTLINE"
    # only show the line
    #echo $CERTLINE || die "Failed to get the certficate with: $CERTLINE"
    
    
    # link it to services
    if [ "$(postconf)" ]; then
    	# pull in the postconf variables
    	source <(postconf smtpd_tls_cert_file smtpd_tls_key_file | tr -d ' ')
    	# remove the old cert and key (!) and link to the new ones
    	rm -f $smtpd_tls_cert_file $smtpd_tls_key_file
    	ln -s /etc/letsencrypt/live/$ISPSERVER/fullchain.pem $smtpd_tls_cert_file
    	ln -s /etc/letsencrypt/live/$ISPSERVER/privkey.pem $smtpd_tls_key_file
    	# restart services
    	service postfix reload
    	service dovecot reload
    fi
    
    
     
    ahrasis and till like this.
  20. vincentkersten

    vincentkersten New Member

    Restarting the services when a cert has been renewed. Although the solution of Jesse is correct, it seems also a little heavy. Here is a 'local' alternative which compares the age of the postfix/dovecot process and the age of the letsencrypt certificate.

    Code:
    #!/bin/bash
    
    # Cronjob for root:
    # 1 * * * * /usr/local/ispconfig/server/restart-mail-services.sh | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    
    # compare runtime of postfix/dovecot with age of LE-certificate
    # to restart these services
    
    # get the domain of the host
    ISPSERVER=$(hostname -f)
    
    # default location of the cert (symlink)
    CERT=/etc/letsencrypt/live/$ISPSERVER/cert.pem
    
    # die with err.
    die () {
    	echo "ERROR: $1"
    	exit 1
    }
    
    function need_restart {
    	# check if the process is running
    	if [ $(pidof $1) ]; then
            	# Get the age of process in seconds
    		PROC_AGE=$(ps -o etimes= -p $(pidof $1))
    		# compare age of process to age of cert and restart
            	if [ $PROC_AGE -gt $CERT_AGE ]; then
            	       service $2 restart &>/dev/null || echo "Could not restart $2"
            	fi
    	fi
    }
    
    # Check if letsencrypt cert is found or fail
    [ -f $CERT ] || die "Letsencrypt certificate ($CERT) not found. Please set up letsencrypt!"
    
    # Calculate cert age in seconds (curr date - epoch date of cert)
    CERT_AGE=$(( $(date +%s) - $(stat -L -c %Z $CERT) ))
    
    # Check and restart if needed.
    # postfix main process is called master, dovecots' is dovecot
    need_restart master postfix
    need_restart dovecot dovecot
    
    exit 0
    
    
     
    Jesse Norell likes this.

Share This Page