ISPconfing 3 - stopped after server restart

Discussion in 'General' started by itstudio7777, Mar 23, 2022.

  1. itstudio7777

    itstudio7777 Member

    Hello, I have very strange problem.

    My VPS is Debian 10 with Apache2 and ISPconfig 3 installed on it.
    It worked about 4 months without problems, but today I have server overload and I make restart of mine VPS.
    After that my ISPConfig is not working I did NOT make changes or settings, but I can't connect to my websites and also to ISPconfig admin panel.


    Please, help me to resolve this problem, I don't know what's going on
     
  2. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Can you acces the server over SSH?
     
  3. itstudio7777

    itstudio7777 Member

    yes, I can.
    I also can connect to Ip/phpmyadmin/ but it don't take my password and user

    Code:
     Cannot log in to the MySQL server
    
     mysqli_real_connect(): (HY000/2002): No such file or directory
     Connection for controluser as defined in your configuration failed.
     mysqli_real_connect(): (HY000/2002): No such file or directory
     
  4. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Have you tried rebooting the full server?
     
  5. itstudio7777

    itstudio7777 Member

    yes, I have tried to reboot, tried to restore from WORKING archive but problem is not resolving.

    In my /etc/phpmyadmin I have only htpasswd.setup , I don't have config file.
    I can't start MariaDB and I can't access to ISPconfig - it's not active
     
  6. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    What is the output of
    Code:
    systemctl status apache2
    and
    Code:
    systemctl status mariadb
    ?
     
  7. itstudio7777

    itstudio7777 Member

    Code:
    Mar 23 14:23:20 server1 systemd[1]: Started The Apache HTTP Server.
    Mar 23 14:24:11 server1 phpMyAdmin[719]: user denied: root (mysql-denied) from 85.187.27.2
    
    Code:
    Mar 23 14:23:19 server1 systemd[1]: Starting MariaDB 10.3.31 database server...
    Mar 23 14:23:19 server1 sh[508]: mktemp: failed to create file via template ‘/tmp/wsrep_recovery.XXXXXX’: Read-only file system
    Mar 23 14:23:19 server1 sh[508]: WSREP: mktemp failed
    Mar 23 14:23:19 server1 systemd[574]: mariadb.service: Failed to set up mount namespacing: Read-only file system
    Mar 23 14:23:19 server1 systemd[574]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: Read-only file system
    Mar 23 14:23:19 server1 systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
    Mar 23 14:23:19 server1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
    Mar 23 14:23:19 server1 systemd[1]: Failed to start MariaDB 10.3.31 database server.
    
     
  8. itstudio7777

    itstudio7777 Member

    I tried also to stop the server and then to start it.
    After that I can’t start Apache service.
    My error is “result: Resources” now I don’t have running Apache and MySQL on my server.

    I really don’t know what to do
     
  9. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    You have a lot of 'Read-only file system' errors, your root filesystem (and/or other filesystems) may need a check/repair. This is done from the server console, either boot into single user mode or boot off a rescue medium and run a filesystem check (the exact command depends on the filesystem type and device(s) to be checked).
     
  10. itstudio7777

    itstudio7777 Member

    yes, you are right !
    I have checked with journalctl -xe and there are a lot files 'read only' and not writeable.
    I have checked the command /etc/fstab but there is an error :

    Code:
    -bash: /etc/fstab: Permission denied
    
    then I tried nano /etc/fstab and it showed me list and at the end of list is :
    [ File '/etc/fstab' is unwritable ]

    What can I do to repair my server?
     
  11. itstudio7777

    itstudio7777 Member

    I have started Apache2 with mount / umount commands but I can't start MariaDB now.

    I have this error when trying to start with command 'sudo service mariadb start' :

    Code:
    -- The unit mariadb.service has entered the 'failed' state with result 'signal'.
    Mar 23 18:32:31 server1 systemd[1]: Failed to start MariaDB 10.3.31 database server.
    -- Subject: A start job for unit mariadb.service has failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- A start job for unit mariadb.service has finished with a failure.
    --
    -- The job identifier is 3684 and the job result is failed.
    Mar 23 18:32:31 server1 sudo[3025]: pam_unix(sudo:session): session closed for user root
    Mar 23 18:32:31 server1 sshd[3023]: Failed password for root from 81.68.167.224 port 60892 ssh2
    Mar 23 18:32:32 server1 sshd[3023]: Connection closed by authenticating user root 81.68.167.224 port 60892 [preauth]
    Mar 23 18:32:34 server1 sshd[3110]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=81.68.167.224  user=root
     
  12. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Reboot the server. When it starts, make it run file system checks.
    On Debian 10, issue command
    Code:
    touch /forcefsck;
    More info from Internet Search Engines with
    Code:
    debian 10 force file system check at boot
    You may have to do that a second time, first check may not find and repair all errors.
     
  13. itstudio7777

    itstudio7777 Member

    I made it and the problem is again there :

    Code:
    Mar 23 19:08:52 server1 sudo[2354]: pam_unix(sudo:session): session closed for user root
    Mar 23 19:08:52 server1 sshd[2350]: Connection closed by authenticating user root 81.68.167.224 port 46674 [preauth]
    Mar 23 19:08:54 server1 sshd[2439]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=81.68.167.224  user=root
    Mar 23 19:08:56 server1 sshd[2439]: Failed password for root from 81.68.167.224 port 47824 ssh2
    
     
  14. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did the 'Read-only file system' errors disappear?
     
  15. itstudio7777

    itstudio7777 Member

    Yes, 'read-only file system' is not present error.
    Now can't start MySQL (MariaDB)

    When I type 'sudo service mysql start' I'm receiving :

    Code:
    Job for mariadb.service failed because a fatal signal was delivered to the control process.
    
    and in system information I have this one:
    Code:
    ● mariadb.service - MariaDB 10.3.31 database server
    [INDENT][INDENT]   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/mysql.service.d
               └─limits.conf
       Active: activating (auto-restart) (Result: signal) since Wed 2022-03-23 19:52:58 GMT; 4s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 3406 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
      Process: 3407 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 3408 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && syst
      Process: 3453 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, signal=SEGV)
     Main PID: 3453 (code=killed, signal=SEGV)
       Status: "Starting final batch to recover 17 pages from redo log"
    

    I think the problem is here : main PID: 3453 (code=killed, signal=SEGV) but I don't know what is this and how to fix it
     
  16. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Just guessing, you may have corruption in some mysql file(s). You could back them up first if you want (eg. make a copy of everything under /var/lib/mysql), then run use mysqlcheck to attempt to repair all databases (a quick search says that should be "mysqlcheck -c -u root -p --all-databases"). That may not be the issue, you could have other corruption in a binary or library, etc.
     
  17. itstudio7777

    itstudio7777 Member

    I also think so, after searching of information but I don’t want to lose nothing from my information in databases.
    If I copy them, like you said, will I lose information ?
     
  18. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    For the most part, short of some data recovery feats which are above my pay grade, any data loss/corruption you have at this point is already done. If you copy all the mysql stuff before repairing it, then at least any inconsistencies the repair would "fix" and result in data loss can be reverted back to the point at which you are now. That may not do you any good, but is at least provides a measure of "lets try that again a different way."

    If you think the data is so valuable you might spend $thousands on data recovery in a worst case, you should stop running the server now so that no more changes are made to the disk, then clone the disk and work from the clone. But if it's just you doing a best effort recovery, make a copy, try the repair, and see where you're at. If things don't go well down that road, hopefully you at least have your most recent set of backups to work from. In a best case, some inconsistencies are repaired and you continue on without any issue whatsoever.
     
  19. itstudio7777

    itstudio7777 Member

    you are right, again !

    So, I tried this : mysqlcheck -c -u root -p --all-databases and server got error :

    Code:
    [email protected]:~# mysqlcheck -c -u root -p --all-databases
    Enter password:
    mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect
    [email protected]:~#
    I can't connect to local MySQL server , but don't understand why
     
  20. itstudio7777

    itstudio7777 Member

    Now problem is with ISPConfig starting :

    Code:
    Mar 23 23:10:07 server1 mysqld[4271]: 2022-03-23 23:10:07 218 [Warning] Access denied for user 'ispconfig'@'localhost' (using password: YES)
    Mar 23 23:10:07 server1 mysqld[4271]: 2022-03-23 23:10:07 219 [Warning] Access denied for user 'ispconfig'@'localhost' (using password: YES)
    Mar 23 23:10:11 server1 mysqld[4271]: 2022-03-23 23:10:11 220 [Warning] Access denied for user 'ispconfig'@'localhost' (using password: YES)
    
     

Share This Page