recovery from failed update

Discussion in 'Installation/Configuration' started by chief, Jun 26, 2020.

  1. chief

    chief Member

    Hi,
    I updated my mail server today, after a long time for a reboot, i investigated it and i had a grub prompt up. i have tried to get the system to boot and it wont, either goes into busybox prompt if i select a previous kernel or if i choose latest it crashes.
    So, i have it in recovery off a loaded cd disk, on lan with drive mounted. i have another identical raid server install debian as i type.
    Can i just install needed binaries following setup of perfect debian server and then scp all the directories containing ispconfig files??
    as it is a mail server, it only has /var/vmail/.... etc
    And i need it up and running like 2 hours ago.. (well chief, you should have made time for the backup cloning of primary mail server you were planning for last 6 months) yes yes i know!!

    please help obi wan
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    First find out what OS is running on that host.
    You wrote
    but please tell what kind of update? Just apt update or apt dist-upgrade and update of what, the OS, ISPConfig or both or something else?
    Did you get errors or warnings during the update? What errors do you get now that booting fails? Try finding out what is wrong, then fix that.
    There is this Tutorial, I would try this or other similar documents first to repair the boot:
    https://www.howtoforge.com/tutorial/repair-linux-boot-with-grub-rescue/
    Read also the comments, they usually have good info on the tutorial.
    Assuming the update you did installed a working kernel, you should get back to a successful boot by reconfiguring grub to boot the right kernel with the approprieate settings. Or a previous kernel.
     
  3. chief

    chief Member

    confimation - multi server setup, debian 9.12, ispconfig 3.1.15 - web, 2 x dns, 1 mail server - the mail server is the issue
    All i did was
    Code:
    apt-get update
    then
    Code:
    apt-get upgrade
    , i rebooted the server and it came to stop at grub-rescue.
    I have followed your link already, didn't fix my issue.
    Code:
    ls
    - this lists (hd0,gpt2) as my ext partition
    i try
    Code:
    set prefix=(hd0,gpt2)/boot/grub
    Code:
    set root=(hd0,gpt2)
    Code:
    insmod normal
    - this gives me error: attempt to read or write outside of disk (hd0)
    and this is odd..
    Code:
    ls (hd0,gpt2)/
    - this lists all directories
    Code:
    ls (hd0,gpt2)/boot
    - gives error attempt to read or write outside of disk (hd0)

    So, i have another same dell R710 server 3 x 2TB SAS drives RAID 5, which i installed debian 9.12, all goes well until reboot and gives same error. i have tried again and again - same result.
    I have done same install using debian 10 - also same result.
    all attempts end in grub-rescue and cannot boot, i try and pointgrub using set command to kernel .

    So,for now i have installed debian 9.12 on a singleHD machine (working) to just get the mail server running again, i want to compress the ispconfig 3 from old system mounted using rescue disk.. (its compressing /var/vmail now-
    what directories in total do i need to move (compress) to reinstate my mail server..
    /var/vmail/, /usr/local/ispconfig/, /etc/passwd/, /etc/group/, /var/lib/mysql and any other directories that ispconfig used to run
     
  4. ahrasis

    ahrasis Well-Known Member

    I would advise you to access and backup the server's contents (file, database, config etc) as there could be hdd failure or more, and only once that is done, then try to revive it.
     
  5. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    So you have three same model machines that all fail in the same fails to boot way? This makes me think the new kernel or grub is at fault, or maybe driver for raid card or something. Have you checked debian bug reports? Or forums for dell R710 server?
    Did you previously install some driver or kernel module from outside Debian on those hosts?
     
    chief likes this.
  6. chief

    chief Member

    No, different dell raid server for web / control panel - that works.
    But on Dell R710, that machine has 4 x 2TB all RAID 5 together - this is the mail server. This is the one that refuses to load. all files are there, just having errors when accessing /boot/..... that tells me
    Code:
    error attempt to read or write outside of disk (hd0)
    it is also the same when i attempt a fresh install on spare server, finish install reboot then i get the same error.
    Its all very odd.
    Anyway, my main concern is to move the ispconfig and all the mail dirs to another pc..
    can you please reply with the exact directories i can drop on to another standard PC thattemporarily will become mail server.

    thanks
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Search the forum for threads about migrating ispconfig manually, e.g.:

    https://www.howtoforge.com/community/threads/ispconfig-3-move-to-another-server.41643/
    https://www.howtoforge.com/community/threads/upgrade-ubuntu-ispconfig-3-from-12-04-14-04.66552/
     
  8. chief

    chief Member

    Thanks Till,
    As i only have old server in recovery mode, i added
    Code:
    /var/lib/mysql/*
    and compressed the databases then sent to replacement server. I have rebooted the system and mail is acepting people loggin in, but have issues.
    I have tried ispconfig_update.sh it errors at the end with
    Code:
    Checking ISPConfig database .. OK
    PHP Fatal error:  Uncaught Error: Call to undefined function mb_detect_encoding() in /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php:448
    Stack trace:
    #0 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php(166): db->escape('validate_passwo...')
    #1 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php(231): db->_build_query_string('SELECT plugin_n...', Array)
    #2 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php(270): db->_query('SELECT plugin_n...', Array)
    #3 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php(319): db->query('SELECT plugin_n...', Array)
    #4 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/update.lib.php(139): db->queryAllRecords('SELECT plugin_n...', Array)
    #5 /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/update.php(315): updateDbAndIni()
    #6 {main}
      thrown in /tmp/update_stable.sh.sA0nJZ1vyM/ispconfig3_install/install/lib/mysql.lib.php on line 448
    seeing 1st line, google it and i get a result install
    Code:
    php-mbstring
    Code:
    The following NEW packages will be installed:
      php-mbstring php7.0-mbstring
    0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
    Need to get 0 B/470 kB of archives.
    After this operation, 1,522 kB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    dpkg: unrecoverable fatal error, aborting:
     unknown group 'Debian-exim' in statoverride file
    E: Sub-process /usr/bin/dpkg returned an error code (2)
    An update / edit
    look like a user in
    Code:
    /var/lib/dpkg/statoverride
    i have backed up the file and install php-mbstring and it looks like it updating
     
    Last edited: Jun 27, 2020
  9. chief

    chief Member

    new error
    Code:
    Jun 27 12:21:43 mail postfix/postdrop[438]: warning: mail_queue_enter: create file maildrop/816459.438: Permission denied
    changed permissions
    Code:
    chown -R postfix:postfix /var/spool/postfix
    not displaying permission error now.
    Code:
    postfix check
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/.
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/pid
    postfix/postfix-script: warning: not owned by postfix: /var/lib/postfix/.
    postfix/postfix-script: warning: not owned by postfix: /var/lib/postfix/./master.lock
    postfix/postfix-script: warning: not owned by group postdrop: /var/spool/postfix/public
    postfix/postfix-script: warning: not owned by group postdrop: /var/spool/postfix/maildrop
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/etc
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/etc/ssl
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/etc/ssl/certs
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/etc/ssl/certs/ca-certificates.crt
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/lib
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/lib/x86_64-linux-gnu
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/usr
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/usr/lib
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/usr/lib/sasl2
    postfix/postfix-script: warning: not owned by root: /var/spool/postfix/usr/lib/zoneinfo
    is this alright?
    Fixed it..
    Ok,
    Code:
    postfix check
    reported incorrect permissions. checking my old system shows different permissions and some sub folders owned by postfix and postgrey with some sub folders with a variety of both, went though setting permissions and now sorted no more errors in /var/log/mail.log and the postfix mailcheck is clear.
    sending emails again.
    thank you all for the point in to right direction
     
    Last edited: Jun 27, 2020

Share This Page