Scheduled backups stopped working

Discussion in 'General' started by bpmee, Apr 18, 2022.

  1. bpmee

    bpmee Member

    Just noticed today that scheduled backups stopped working. The last backup was several weeks ago, about the time I upgraded ]# ispconfig_update.sh. To be fair, I'm not sure if this was the cause.

    Initial Troubleshooting:
    1. Running a website manual backup works.
    2. Executing from command line seems to work, no errors ]# php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
    3. ]# df -h shows plenty of disk space.
    4. Tried changing backup time from 01:30 to 17:30. Switched it back to 01:30 for the next day, it did not run.
    5. Installed BorgBackup even though I do not use it, just in case that was preventing standard backups.
    6. All other CronJobs run normally, no other disruptions that I can see.
    Server Environment:
    CentOS 7.9, ISPConfig 3.2.8p1

    Maybe I told ispconfig_update.sh to reconfigure services last update, and this affected the backups?

    Thanks for any ideas!
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  3. bpmee

    bpmee Member

    Hi Taleman,

    Thanks for your reply. I haven't debugged ISPConfig in a while a completely forgot about the cron debugging feature.

    1. I set the log level to Debug as instructed.
    2. I set the backups to run in a hour. Will see what comes up in the log.
    Here are the results from Till's common issues script:

    Code:
    wget -q -O htf-common-issues.php "http://gitplace.net/pixcept/ispconfig-tools/raw/stable/htf-common-issues.php" && php -q htf-common-issues.php
    
    cat htf_report.txt | more
    
    Code:
    ##### SCRIPT FINISHED #####
    Results can be found in htf_report.txt
    To view results use your favourite text editor or type 'cat htf_report.txt | more' on the server console.
    
    If you want to see the non-anonymized output start the script with --debug as parameter (php -q htf-common-issues.php --debug).
    
    [[email protected] home]# cat htf_report.txt | more
    
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is "CentOS Linux release 7.9.2009 (Core)"
    
    [INFO] uptime:  14:21:38 up 76 days, 12:42,  1 user,  load average: 0.67, 0.55, 0.62
    
    [INFO] memory:
                  total        used        free      shared  buff/cache   available
    Mem:            31G        7.5G        1.0G        1.7G         22G         21G
    Swap:           15G         48M         15G
    
    [INFO] systemd failed services status:
      UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
    ● cdp-agent.service              loaded failed failed LSB: Starts R1Soft CDP Agent
    ● systemd-tmpfiles-clean.service loaded failed failed Cleanup of Temporary Directories
    
    LOAD   = Reflects whether the unit definition was properly loaded.
    ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
    SUB    = The low-level unit activation state, values depend on unit type.
    
    2 loaded units listed. Pass --all to see loaded but inactive units, too.
    To show all installed unit files use 'systemctl list-unit-files'.
    
    [INFO] ISPConfig is installed.
    
    ##### ISPCONFIG #####
    ISPConfig version is 3.2.8p1
    
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 7.4.16
    [INFO] php-cgi (used for cgi php in default vhost!) is version 7.4.16
    
    ##### PORT CHECK #####
    
    [WARN] Port 22 (SSH server) seems NOT to be listening
    
    ##### MAIL SERVER CHECK #####
    
    [WARN] I found no "submission" entry in your postfix master.cf
    [INFO] this is not critical, but if you want to offer port 587 for smtp connections you have to
    enable this.
    
    ##### RUNNING SERVER PROCESSES #####
    
    [INFO] I found the following web server(s):
            Unknown process (httpd) (PID 29277)
    [INFO] I found the following mail server(s):
            Postfix (PID 12564)
    [INFO] I found the following pop3 server(s):
            Dovecot (PID 12631)
    [INFO] I found the following imap server(s):
            Dovecot (PID 12631)
    [INFO] I found the following ftp server(s):
            PureFTP (PID 12880)
    
    ##### LISTENING PORTS #####
    (only           ()
    Local           (Address)
    [anywhere]:993          (12631/dovecot)
    [anywhere]:995          (12631/dovecot)
    [localhost]:10024               (12592/amavisd)
    [localhost]:10025               (12564/master)
    [localhost]:10026               (12592/amavisd)
    [localhost]:10027               (12564/master)
    [anywhere]:15531                (1137/sshd)
    [anywhere]:110          (12631/dovecot)
    [anywhere]:143          (12631/dovecot)
    [anywhere]:1167         (4212/cdp)
    [anywhere]:465          (12564/master)
    ***.***.***.***:53              (12905/named)
    ***.***.***.***:53              (12905/named)
    ***.***.***.***:53              (12905/named)
    [localhost]:53          (12905/named)
    [anywhere]:21           (12880/pure-ftpd)
    [localhost]:953         (12905/named)
    [anywhere]:25           (12564/master)
    [localhost]:8891                (1373/opendkim)
    *:*:*:*::*:993          (12631/dovecot)
    *:*:*:*::*:995          (12631/dovecot)
    *:*:*:*::*:10024                (12592/amavisd)
    *:*:*:*::*:10026                (12592/amavisd)
    *:*:*:*::*:3306         (12434/mysqld)
    [localhost]5531         (1137/sshd)
    [localhost]10           (12631/dovecot)
    [localhost]43           (12631/dovecot)
    *:*:*:*::*:8080         (29277/httpd)
    *:*:*:*::*:80           (29277/httpd)
    *:*:*:*::*:8081         (29277/httpd)
    *:*:*:*::*:465          (12564/master)
    *:*:*:*::*:53           (12905/named)
    *:*:*:*::*:21           (12880/pure-ftpd)
    *:*:*:*::*:953          (12905/named)
    *:*:*:*::*:25           (12564/master)
    *:*:*:*::*:443          (29277/httpd)
    
    
    
    
    ##### IPTABLES #####
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    f2b-FTP    tcp  --  [anywhere]/0            [anywhere]/0            tcp dpt:21
    f2b-dovecot  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 110,995,1
    43,993
    f2b-postfix-sasl  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 25,4
    65,587
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain f2b-FTP (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    
    Chain f2b-dovecot (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    
    Chain f2b-postfix-sasl (1 references)
    target     prot opt source               destination
    REJECT     all  --  ***.***.***.***          [anywhere]/0            reject-with icmp-port-unrea
    chable
    REJECT     all  --  ***.***.***.***        [anywhere]/0            reject-with icmp-port-unreach
    able
    REJECT     all  --  ***.***.***.***          [anywhere]/0            reject-with icmp-port-unrea
    chable
    REJECT     all  --  ***.***.***.***          [anywhere]/0            reject-with icmp-port-unrea
    chable
    REJECT     all  --  ***.***.***.***         [anywhere]/0            reject-with icmp-port-unreac
    hable
    REJECT     all  --  ***.***.***.***          [anywhere]/0            reject-with icmp-port-unrea
    chable
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    
    
    
    
    ##### LET'S ENCRYPT #####
    Certbot is installed in /usr/bin/letsencrypt
    
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    I may be wrong, but I don't think the schedule works like that, there is a "next backup cron job" scheduled to run at the moment, and when that runs (at 1:30am it sounds like) it will set the time for the next job according to the time it is configured to run at.
     
    Last edited: Apr 19, 2022
  5. bpmee

    bpmee Member

    Ah...you may be right. So I'd have to wait another 24 hours for the next one to trigger?

    Here are my settings:
     

    Attached Files:

  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes
     
  7. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Did you mean Uninstalled?
    Is that the correct version for CentOS?
     
  8. michelangelo

    michelangelo Member

    No. The version that is shipped with CentOS 7 is PHP 5.4.16.
    The installed PHP version is also not the most recent version of the PHP 7.4 branch.

    As long as the PHP version was correctly upgraded/installed I don't see any issues, although it is not recommended to upgrade the base PHP version... The question remains though, how the actual situation looks like on the server.

    So, how or from which repository (or software collection) was this PHP version installed and what are the ispconfig.log and especially the cron.log files saying when the OP runs the backup via cron_debug.php? @bpmee
     
  9. bpmee

    bpmee Member

    Hi @michelangelo

    1. I upgraded PHP 1+ years ago because some software (Wordpress, and some plugins) were complaining about PHP 5.x. Both ISPConfig and backups ran fine after the update. The backup problem started in the past few weeks.

    2. I believe it was EPEL repo and REMI repos? As I said, no major problems after update. This is a common thing to do as PHP 5 is no longer supported as of 2019.

    3. I can send ]# php -i output if you need it.

    Hi @Taleman - I was able to install BorgBackup successfully on my system. Again, I don't use it. ISPConfig warned it was missing. I thought this might be causing backups stop running.

    Hi @all

    I scheduled the backup to run at a different time yesterday (15:30) to test. Looks like it worked.

    Immediately following that result, I rescheduled it to 05:00 (normal). I'm waiting for that event to trigger tomorrow morning, as it takes 24 hours following what @Jesse Norell and @till advised. Will see if it runs.

    Question: Any chance the backup script conflicts with something running around 05:00?
     
  10. michelangelo

    michelangelo Member

    This is actually not common what you did since the best-practice way is to set up a newer PHP version for ISPConfig under "Additional PHP Versions" and leave the base PHP version with what it was initally shipped. The distribution maintainer always takes care of the security updates and will apply them as necessary.

    Upgrading the base PHP version is ofc generally possible but only adviseable if you know what you might cause with such actions.
     
    bpmee likes this.
  11. bpmee

    bpmee Member

    @michelangelo Thanks for your expertise. :) I didn't know about that feature.
    I followed some tutorials I found elsewhere. I was lucky the upgrade didn't cause any additional problems.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    It will run tomorrow at the same time it ran today and the day after tomorrow it will run at 05:00. The next start time is set when the cronjob runs, so when you change the time after the current run of a cronjob starts, the change will get applied after the next run.

    Changing the base PHP version on centOS is less critical than changing it on Debian and ubuntu as CentOS puts every PHP version in the same path, so there will be no issues unless you choose a PHP version > 7.4 on CentOS.
     
    bpmee likes this.
  13. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    So @bpmee, note that out of a concern for using outdated/insecure php versions, you installed a specific 7.4.x point release which is now a year out of date and has many known security vulnerabilities. You should ensure all your php versions stay updated (I'm not familiar enough with centos to give specific recommendations in doing so).
     
  14. bpmee

    bpmee Member

    @till, @jesse Norwell - Thanks very much.
    I did a ]# yum update php and nothing was marked for update (yet). I periodically check my packages. Luckily no security issues (yet).

    Will monitor backups over the next day. I now understand that backup time is set with each cron job, firing the backup at the new time after the existing day's job.
     
  15. bpmee

    bpmee Member

    Hi, scheduled backups for 05:00 didn't run. I don't see any errors at that time in /cron.log or /ispconfig.log.

    Any ideas?
     
  16. bpmee

    bpmee Member

    @Taleman @Jesse Norell
    Scheduled backups still aren't running.

    I don't see any errors when I run the debugging script from the command line:
    Code:
    ]# php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
    Can I run ISPConfig's actual backup script from the command line and watch it "live" for any errors?
    Very strange as the rest of the server is fine. There are no alerts in the Monitoring panel.
     
  17. bpmee

    bpmee Member

    SOLVED! - Crontab was missing the ISPConfig Cron task.
    Code:
    * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done
    
    The above must be present in addition to the /server.sh job. I initially missed it because standard debugging didn't return any errors. Also, ISPConfig's Monitor had no errors or warnings.

    Backups are running nicely again. Thanks @till, @Taleman @Jesse Norell , @michelangelo for your assistance!

    So everything was fine **except** for the cronjob entry on the root crontab.

    Finally, I researched potential causes. I don't think it was any ISPConfig update. Rather, I may have removed the cronjob by mistake or some other script trimmed it off.

    I've been using ISPConfig since 2008. While I'm not an "expert" user, I've successfully managed my own server over the years. To date, no major problems, security issues, or hacks. It's very stable and reliable! :)
     
    Th0m and till like this.

Share This Page