Websites have stopped responding, how to troubleshoot?

Discussion in 'General' started by schwim, Dec 7, 2020.

  1. schwim

    schwim Member

    Hello everyone!
    Today, I tried to add a new website to the ISPC control panel and was met with very slow page load times. When I tried to visit one of the sites hosted on the machine, it resulted in a non-responsive website error. Testing with an independent service confirmed that the site was not responsive.

    After a server reboot, I was no longer able to even access the ISPC control panel. I can still shell in as root or user and my SSHFS connections are still working. I just can't access the websites on the machine, including ISPC control panel.

    Where should I begin troubleshooting this issue?

    Thanks!
    Jason
     
  2. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Is your disk full? How much ram do you have, is there enough available?
    What is the output of
    Code:
    systemctl list-units --failed
     
  3. schwim

    schwim Member

    Hello there and thanks for the help!
    [​IMG]
    [​IMG]
    [​IMG]
     
  4. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    That looks bad ;)
    Take a look at the status of those failed daemons, for example apache2, with
    Code:
    systemctl status apache2
    and see if there are any errors pointing you in the right direction. Check your logs in /var/log aswell (for example syslog)
     
  5. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I don't know about your websites space but I'd normally look update (if any) and test my nginx web server and if there is nothing wrong, I'll check my php-fpm for all versions. For php-fpm I normally ensure all the needed and latest Ondrej Sury versions are installed / updated which now include php-8.0. I ensure my nginx web server is also from Ondrej Sury PPA as suggested in his php PPA.
     
  6. schwim

    schwim Member

    It does look bad, indeed :)

    Here's my syslog. I'm a layman and am having problems finding meaningful red lights in the document but hope that you might be able to pull a root issue from it:

    https://schw.im/syslog.txt

    This is quite a pickle in which I find myself. I'm sorry I'm not providing more helpful info but when prompted, I'll do my best to add more info.

    Thanks!
     
  7. schwim

    schwim Member

    I've found thousands of clamav errors in syslog, is there a chance that's causing an issue or is it just a symptom of something upstream?

    I played around with trying to restart apache2 and found some hopefully useful information:

    The "read only filesystem" sounds ominous. Could this be the root of my issue?
     
    Last edited: Dec 8, 2020
  8. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Looks like Apache is starting and running OK.
    You had tons of other services also failing, most importantly mariadb. Test all of those and find out why they are not starting.
     
  9. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    No, the job for Apache failed.

    Can you indeed test the other services to view the details why they failed?

    Did you mess around with file permissions?
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer


    Bad harddisk? Checj if / bis really read only, e.g. with:

    touch /test.txt

    if you get an error that the file can not be created, then your disk is most likely mounted read only. In that case, test and repair the partition with fdsik and then reboot afterward.
     
  11. schwim

    schwim Member

    [email protected]:/var/log# touch /test.txt
    touch: cannot touch '/test.txt': Read-only file system

    It 's a read only filesystem. I wanted to look at /etc/fstab and found this:

    That dogwoodthicket path is the last site I added to ISPC and doing that somehow wiped my fstab, there are no other lines in that file.

    How do I go about restoring the mount lines in fstab?
     
  12. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Do you find any backup copies, eg named fstab~ or fstab.bak ? If not, check your recent backups, including the one ISPconfig offers to make when you upgrade.
     
  13. schwim

    schwim Member

    Unfortunately, I have no backup for the file. Nothing in /etc and I've not upgraded ISPC on this VPS. I'm pretty sure I'm going at it from scratch.
     
  14. schwim

    schwim Member

    Looking at this:
    [​IMG]

    Could I just use this to mount /dev/xvda2 to / ?

    Code:
    /dev/xvda2     /               ext4    errors=remount-ro 0       1
    
     
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    You can try it, or better ask the support of your vps provider if they can give you the default content of the fstab file for their vm's, as this can differ a lot.
     
  16. schwim

    schwim Member

    I asked and will see what they say but for now, I figured I'd just steal my fstab mounting line from my home debian install and I'm back in a writeable OS. Unfortunately, most of the services are still failing to start. As an example, here's mariadb(from syslog):
    The "no such process" seems worrisome. I looked in /usr/bin and there is indeed the "install" process. Does the error point toward another issue I need to pursue like the read-only error did?
     
    Last edited: Dec 8, 2020
  17. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    /proc/mounts might be helpful as well, I'd at least save a copy now before you reboot.
     
  18. schwim

    schwim Member

    thanks Jesse, I've got the file system mounting writable, am now trying to sort all of the failed starting processes. See my post above yours for the first one I'm trying to tacckle(mariadb). I've searched for "Failed to determine group credentials: No such process" and "Failed at step GROUP spawning /usr/bin/install: No such process" but all I find in Google are results relating to systemd and other errors not quite matching.
     
  19. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Try to install monit to monitor your server services in the future as it can send emails alerting you even if you forget to check manually. It also can viewed inside ISPConfig but you will need to add "use address 0.0.0.0" and enable ssl in monitrc file.
     
  20. schwim

    schwim Member

    I'm still struggling to get this machine up and running again. None of the processes ISPC require are operating and after days of Googling, the only thing I can come up with is that the groups required by systemd aren't being found:

    As an example, here’s mariadb:

    Dec 8 09:03:39 adolf systemd[473]: mariadb.service: Failed to determine group credentials: No such process
    Dec 8 09:03:39 adolf systemd[473]: mariadb.service: Failed at step GROUP spawning /usr/bin/install: No such process
    Dec 8 09:03:39 adolf systemd[1]: mariadb.service: Control process exited, code=exited, status=216/GROUP
    Dec 8 09:03:39 adolf systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Dec 8 09:03:39 adolf systemd[1]: Failed to start MariaDB 10.3.22 database server.
    Failed to determine group credentials: No such process
    Failed at step GROUP spawning /usr/bin/install: No such process

    These groups clearly existed for nearly a year as the server has worked without issue but I don't know how to determine which groups are needed, how to see which currently exist and how to add the required groups. Is this something that can be managed or is this irrecoverable?
     

Share This Page