Apache reloading loop when renewing SSL Certs (acme.sh)

Discussion in 'General' started by Brad Trammell, Jan 19, 2022.

  1. Brad Trammell

    Brad Trammell New Member

    We are seeing an issue on one of our ISPConfig 3 servers that when acme.sh renews, it causes httpd to get into a reloading loop where basically the apache service freezes up while reloading, and acme.sh times out trying to renew or verify the order.

    Here is the output when running the command manually:
    [11:19] [server acme.sh] # acme.sh --issue --apache -d example.com -d www.example.com --force
    [Wed Jan 19 11:20:03 EST 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory
    [Wed Jan 19 11:20:04 EST 2022] Checking if there is an error in the apache config file before starting.
    [Wed Jan 19 11:20:04 EST 2022] OK
    [Wed Jan 19 11:20:04 EST 2022] JFYI, Config file /etc/httpd/conf/httpd.conf is backuped to /root/.acme.sh/httpd.conf
    [Wed Jan 19 11:20:04 EST 2022] In case there is an error that can not be restored automatically, you may try restore it yourself.
    [Wed Jan 19 11:20:04 EST 2022] The backup file will be deleted on success, just forget it.
    [Wed Jan 19 11:20:04 EST 2022] Creating domain key
    [Wed Jan 19 11:20:04 EST 2022] The domain key is here: /root/.acme.sh/example.com/example.com.key
    [Wed Jan 19 11:20:04 EST 2022] Multi domain='DNS:example.com,DNS:www.example.com'
    [Wed Jan 19 11:20:05 EST 2022] Getting domain auth token for each domain
    [Wed Jan 19 11:20:06 EST 2022] Getting webroot for domain='example.com'
    [Wed Jan 19 11:20:06 EST 2022] Getting webroot for domain='www.example.com'
    [Wed Jan 19 11:20:06 EST 2022] Verifying: example.com
    [Wed Jan 19 11:20:07 EST 2022] Pending, The CA is processing your order, please just wait. (1/30)
    [Wed Jan 19 11:20:10 EST 2022] Pending, The CA is processing your order, please just wait. (2/30)
    [Wed Jan 19 11:20:12 EST 2022] Pending, The CA is processing your order, please just wait. (3/30)
    [Wed Jan 19 11:20:14 EST 2022] Pending, The CA is processing your order, please just wait. (4/30)
    [Wed Jan 19 11:20:17 EST 2022] Pending, The CA is processing your order, please just wait. (5/30)
    [Wed Jan 19 11:20:19 EST 2022] Pending, The CA is processing your order, please just wait. (6/30)
    [Wed Jan 19 11:20:22 EST 2022] Pending, The CA is processing your order, please just wait. (7/30)
    [Wed Jan 19 11:20:24 EST 2022] Pending, The CA is processing your order, please just wait. (8/30)
    [Wed Jan 19 11:20:27 EST 2022] Pending, The CA is processing your order, please just wait. (9/30)
    [Wed Jan 19 11:20:29 EST 2022] Pending, The CA is processing your order, please just wait. (10/30)
    [Wed Jan 19 11:20:32 EST 2022] Pending, The CA is processing your order, please just wait. (11/30)
    [Wed Jan 19 11:20:34 EST 2022] Pending, The CA is processing your order, please just wait. (12/30)
    [Wed Jan 19 11:20:36 EST 2022] Pending, The CA is processing your order, please just wait. (13/30)
    [Wed Jan 19 11:20:39 EST 2022] Pending, The CA is processing your order, please just wait. (14/30)
    [Wed Jan 19 11:20:41 EST 2022] Pending, The CA is processing your order, please just wait. (15/30)
    [Wed Jan 19 11:20:44 EST 2022] Pending, The CA is processing your order, please just wait. (16/30)
    [Wed Jan 19 11:20:46 EST 2022] Pending, The CA is processing your order, please just wait. (17/30)
    [Wed Jan 19 11:20:48 EST 2022] Pending, The CA is processing your order, please just wait. (18/30)
    [Wed Jan 19 11:20:51 EST 2022] Pending, The CA is processing your order, please just wait. (19/30)
    [Wed Jan 19 11:20:53 EST 2022] Pending, The CA is processing your order, please just wait. (20/30)
    [Wed Jan 19 11:20:56 EST 2022] Pending, The CA is processing your order, please just wait. (21/30)
    [Wed Jan 19 11:20:58 EST 2022] example.com:Verify error:Fetching http://example.com/.well-known/acme-challenge/IXW7pyjMX4a-ogtEAdnR7Gx5PC_-7NFKsR7qtwVOZBk: Connection reset by peer
    [Wed Jan 19 11:20:59 EST 2022] Please check log file for more details: /var/log/ispconfig/acme.log
    Additionally, when checking the log file mentioned, this is what displays.
    https://pastebin.com/mw4ReE3M (had to put it in a PasteBin, it's too long to post)​

    This is essentially happening for all domains on the server, and we can't renew any certs at the moment.

    When it attempts to reload apache it throws the following error:
    Redirecting to /bin/systemctl status httpd.service
    ● httpd.service - The Apache HTTP Server
    Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
    Drop-In: /usr/lib/systemd/system/httpd.service.d
    └─php-fpm.conf, php70-php-fpm.conf, php71-php-fpm.conf, php72-php-fpm.conf, php73-php-fpm.conf, php74-php-fpm.
    conf, php80-php-fpm.conf
    Active: reloading (reload) (Result: core-dump) since Wed 2022-01-19 11:21:20 EST; 12min ago
    Docs: man:httpd.service(8)
    Process: 123496 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
    Process: 54933 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=dumped, signal=ABRT)
    Main PID: 54933 (code=dumped, signal=ABRT)
    Status: "Reading configuration..."
    Tasks: 2 (limit: 49440)
    Memory: 77.4M
    CGroup: /system.slice/httpd.service
    ├─123823 vlogger (access log)
    └─123824 /usr/sbin/httpd -DFOREGROUND

    Jan 19 11:21:20 server httpd[54933]: AH00112: Warning: DocumentRoot [/var/www/server /web] does
    not exist
    Jan 19 11:21:20 server httpd[54933]: [Wed Jan 19 11:21:20.585931 2022] [alias:warn] [pid 54933] AH00671: The
    Alias directive in /etc/httpd/conf/httpd.conf at line 377 will probably never match because it overlaps an earlier Alias
    .
    Jan 19 11:21:20 server systemd[1]: Started The Apache HTTP Server.
    Jan 19 11:21:20 server httpd[54933]: Server configured, listening on: port 8080, port 8081, port 443, port 8
    0
    Jan 19 11:33:42 server systemd[1]: Reloading The Apache HTTP Server.
    Jan 19 11:33:42 server httpd[123496]: AH00548: NameVirtualHost has no effect and will be removed in the next
    release /etc/httpd/conf/httpd.conf:360
    Jan 19 11:33:42 server httpd[123496]: AH00112: Warning: DocumentRoot [/var/www/ispc1.sparkrack.net/web] does
    not exist
    Jan 19 11:33:42 server systemd[1]: Reloaded The Apache HTTP Server.
    Jan 19 11:33:46 server systemd-coredump[123826]: Process 54933 (httpd) of user 0 dumped core.
    Jan 19 11:33:46 server systemd[1]: httpd.service: Main process exited, code=dumped, status=6/ABRT​

    Any assistance I can be given would be great appreciated.
     
    Last edited: Jan 20, 2022
  2. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    You could try 'apachectl -t' and see if it complains of any problems, then maybe 'apachectl -X' to see what else you might find. If that doesn't find anything, my next step would be to run apache under strace and see what you find out.
     
  3. Brad Trammell

    Brad Trammell New Member


    Output of apachectl -t & apachectl -X
    AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:360
    Syntax OK​
     
    Last edited: Jan 20, 2022
  4. Brad Trammell

    Brad Trammell New Member

    Tagging @till for additional assistance.
     
  5. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    In that case, I guess I won't reply.

    Just kidding. But it's nonsense to tag Till for assistance. If you want a specific person to support you, find someone you can hire to give you support. But if you post on the forum, anyone can reply and there are quite some users that help you out here while they aren't part of the ISPConfig team.

    Till reads all threads on this forum (in my experience), so if he wants to reply, he can do that. No need to ping him. But others can help you out just as well. By the way, Jesse already told you what you can try already: run Apache2 under strace, and I would recommend so as well.

    OK, now on to your original post.

    First off, you should never issue a certificate manually, as this will lead to problems with the acme client and the vhost for your website(s). I would recommend deleting the related acme.sh files for this domain and then try enabling it from the UI.

    Alright, so here's a interesting warning already: DocumentRoot [/var/www/ispc1.sparkrack.net/web] does
    not exist
    If you created this site through ISPConfig, that DocRoot should exist.
     
  6. Brad Trammell

    Brad Trammell New Member

    I've posted in the past, and @till and I'm pretty sure you, are usually the only two who answer (for one reason or another), not really sure why. Over the course of the night, this issue has started to affect all domains not just one, as overnight all the certificates on the server expired and are now invalid.

    We generally don't run the command manually, this was taken during debugging after attempts through the UI failed.

    That's the hostname of the server. Which doesn't need a webroot persay. But is supposed to be using the default /var/www/html directory.
     
  7. Brad Trammell

    Brad Trammell New Member

    I think this issue actually goes further than that, because making any changes to domains via the interface results in the following HTTPD crash.

    Jan 20 06:59:09 server systemd-coredump[423457]: Process 73929 (httpd) of user 0 dumped core.
    Jan 20 06:59:09 server systemd[1]: httpd.service: Main process exited, code=dumped, status=6/ABRT
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Still no error when testing the apache config? What does not show up in an apache config test is if SSL cert files are missing, maybe some certs have been deleted?
     
  9. Brad Trammell

    Brad Trammell New Member

    Correct @till. Still no errors in the Apache config. What I find strange is I was just able to issue one via the interface (that previously failed last night) and now that website is showing a 503 error, but apache is running and all of the config for that site is correct (according to the vhost file). But other domains are failing now.
     
  10. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Hm, to further check the SSL files, run the SSL test script (Thanks @Croydon):
    To start the script, run the following command as root user on your server:
    Code:
    curl https://gitplace.net/pixcept/ispconfig-tools/-/raw/stable/cert_check.sh | sh
    Share the output here in code blocks please (insert -> code).
     
  11. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Curious, I must have misunderstood your initial description, I thought apache would not start (as your logs show). What do you do to start it up again, anything manually, or just wait? Or it was stopped, and it is now running again via 'apachectl -X' ?
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Regarding the restart loop, you might want to diable (at least temporarily) the ISPConfig rescue module under System > server config.
     
  13. Brad Trammell

    Brad Trammell New Member

    Command doesn't issue an output, for some reason.

    Apache starts and has no issues until a certificate is requested, once a cert is requested it "reloads" and then stays in that status for about 60-120 seconds, by that time the challenge check has already expired.

    It appears to already be disabled.

    upload_2022-1-21_3-24-42.png
     
  14. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    No output at all?
     
  15. Brad Trammell

    Brad Trammell New Member

    Correct.
     

Share This Page