500 - Internal Server Error

Discussion in 'ISPConfig 3 Priority Support' started by conductive, Nov 2, 2020.

  1. conductive

    conductive Member HowtoForge Supporter

    I am running ispconfig with Nginx on Debian 9.1.3.
    I had a crash and had to run fsck to via KVM to get the OS back and am still getting a 500 Internal server error on webpages and can't access IspConfig.

    cat /var/log/nginx/error.log
    .
    .
    .
    2020/11/01 17:37:36 [error] 1284#1284: *1309 FastCGI sent in stderr: "PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 85
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 85
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP message: PHP Warning: mysqli_real_connect(): (HY000/2002): No such file or directory in /usr/local/ispconfig/interface/lib/classes/db_mysql.inc.php on line 91
    PHP messa
    2020/11/01 17:48:05 [error] 1287#1287: *1514 directory index of "/var/www/apps/" is forbidden, client: 185.172.111.197, server: _, request: "HEAD / HTTP/1.0"

    I am not sure how to recover from this crash. Do I reinstall ISP config or??
    This is not the first time this has happened on this server.
    https://www.howtoforge.com/community/threads/error-500-internal-server-error.83728/
    The server load is typically quite low.

    Thanks for your help
     
    Last edited: Nov 2, 2020
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    What kind of crash? What is the filesystem on the affected partition?
    Forcing ISPConfig update might write the ISPConfig files back on your disk. That might help. You wrote
    but that can not be your Debian version. Do you know what ISPConfig version is on that disk?
     
  3. conductive

    conductive Member HowtoForge Supporter

    Thanks for the reply.

    I guess crashes can be a little vague so all I can do its explain what happened.

    I was having problems with Google SEO and canonical domain names so I started playing around with alias domain names. I started seeing problems with the DNS where many different sites pointed to a went to a single site. Eventually I could not access any pages or ISP config and would get a 500 error. I was able to ssh into the system and perform an update but after a reboot or other I ended up losing the ability to ssh into the server. I had the host set up a KVM and I was able to fsck /dev/mapper/pc--vg-root (ext4) and am now able to ssh into the server but still get 500 errors.

    The Debian version is 9.13

    I am not sure but I believe that I am still running IspConfig 3.1
     
  4. conductive

    conductive Member HowtoForge Supporter

    I attempted to do an update form Ispconfig3.1 to 3.2 and it failed.

    Unable to connect to mysql server No such file or directory
    MySQL root password [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]:
     
  5. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did you have a reason for doing that?
    Did you follow the instructions for updating ISPConfig 3.1 to 3.2?
     
  6. conductive

    conductive Member HowtoForge Supporter

    I did not have a better solution and ...
    "Forcing ISPConfig update might write the ISPConfig files back on your disk. That might help."
    Yes I followed the instructions and updated the proper applications first.
     
  7. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Is mysqld running? You might have to repair tables in order to start it up.
     
    conductive likes this.
  8. conductive

    conductive Member HowtoForge Supporter

    Thanks for the reply...good call ... yes this appears to be a problem
    [email protected]:~# service mysqld status

    ● mariadb.service - MariaDB 10.1.47 database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mysql.service.d
    └─limits.conf
    Active: activating (auto-restart) (Result: signal) since Mon 2020-11-02 14:21:48 CST; 2s ago
    Docs: man:mysqld(8)
    https://mariadb.com/kb/en/library/systemd/
    Process: 2145 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=ABRT)
    Process: 2067 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && sys
    Process: 2063 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 2060 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Main PID: 2145 (code=killed, signal=ABRT)
    Status: "Starting Innodb crash recovery"

    Nov 02 14:21:48 mr1 systemd[1]: mariadb.service: Unit entered failed state.
    Nov 02 14:21:48 mr1 systemd[1]: mariadb.service: Failed with result 'signal'.
     
  9. conductive

    conductive Member HowtoForge Supporter

    Any repair suggestions? This had many word press sites. There are many databases and tables.
     
  10. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Looks like you're using Galera? I'm not familiar with Galera myself, but a quick web search finds some info.
     
  11. conductive

    conductive Member HowtoForge Supporter

    I am not sure what the best approach is.

    [email protected]:~# mysql
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
     
  12. conductive

    conductive Member HowtoForge Supporter

    [email protected]:/etc/mysql# mysqlcheck --auto-repair -o --all-databases -uroot -pXXXXXXXXXXXXXXXXXXXXXX
    mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") when trying to connect

    **** Edit *****
    mariadb is installed
    [email protected]:~# dpkg -s mariadb-server
    Package: mariadb-server
    Status: install ok unpacked

    Any suggestions comments or other?
     
    Last edited: Nov 4, 2020
  13. conductive

    conductive Member HowtoForge Supporter

    Do I need to add/generate a mysqld.sock ???
    What would be the best approach here??
     
  14. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    No, the socket will be created by the db server (or possibly by systemd or similar) when it starts up.

    I believe this is the key phrase to search for a solution:
    However, this makes me wonder if you're running Galera or not:
    The steps to recover with Galera may or may not be different, I don't know.
     
  15. conductive

    conductive Member HowtoForge Supporter

  16. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

Share This Page