Backup failes with "Allowed memory size exhausted"

Discussion in 'ISPConfig 3 Priority Support' started by JanThiel, Jun 3, 2019.

  1. JanThiel

    JanThiel New Member HowtoForge Supporter

    Hi there,

    on a customers server we run the current ISPC version (3.1.13.p1) on a recent and freshly installed CentOS (7.6.1810).
    The server is configured in multi-server mode as a slave and runs the webserver only. The database is running on a separate server.
    We faced issues with backups not being listed in the interface below the "Backups" Tab within the website. The Backup mode was set to "Make backups as root user". Anyway, the backup files were successfully created below /var/backups/web...

    To analyze the issue, we switched to the other Backup mode "Owned by WEB User as ZIP". First that failed due to missing zip and pigz packages. That was fixed. We then run the cron_debug.php script to get more insight. It nearly seemed to run through (the zip file was built below /tmp) but failed with the following errors:

    Code:
    sudo php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
    zip warning: No such file or directory
    ( ^ x >100 )
    .
    .
    zip warning: No such file or directory
    PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /usr/local/ispconfig/server/lib/classes/cron.d/500-backup.inc.php on line 130
    Code:
    php -v
    PHP 7.3.5 (cli) (built: Apr 30 2019 08:37:17) ( NTS )
    Copyright (c) 1997-2018 The PHP Group
    Zend Engine v3.3.5, Copyright (c) 1998-2018 Zend Technologies
    The website we tried to backup is about 30GB huge. The server itself has 32GB RAM + modern Xeon based hardware specs.

    Code:
    tldr;
    - Backup in "root" mode works (zips are created and rotated), but will not show up in the ISPC Admin interface for the websites backups
    - Backup in "user" mode fails with no trace and a memory issue when debugging with the cron_debug.php
    This might be just an issue with the debug tool, but from my understanding, it should not behave any different to the default cron run and the backup should also work with the debug script. I will find out tomorrow after the regular run of the backups tonight.
    Anyway, if there are ideas or known issue, I would be glad to know.

    Thanks alot, best regards,

    Jan
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Try to raise the php memory in php.ini file.
     
  3. JanThiel

    JanThiel New Member HowtoForge Supporter

    Too easy ;-).
    Coming from Debian I expected the CLI memory to not be limited at all. But it seems to be a different story with the remi php packages on CentOS...
    As usual, thanks Till. Let's see how this works out. I will post back.
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    The 'problem' with CentOS compared to Debian is that it uses just a single php.ini file for all kind of PHP use cases while Debian and Ubuntu have a different one for CLI, CGI, mod_php, and FPM which allows more fine-grained control on Debian and Ubuntu.
     
  5. JanThiel

    JanThiel New Member HowtoForge Supporter

    The increased memory partly did the trick.
    Using cron_debug.php it went through this time:
    Code:
    sudo php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
    zip warning: No such file or directory
    ( ^ x 20 )
    zip warning: No such file or directory
    finished.
    But I cannot find any backup files afterwards?!
    Backup mode is still "Zip + User"
    Both folders
    /var/www/[DOMAIN]/backups
    as well as
    /var/backups/webX/
    do not contain any file.

    What should be the expected outcome after the backup cron completes? Should anything be in the /var/www/[DOMAIN]/backups at all?

    Thanks,
    Jan
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, the test script does exactly the same than the normal backup job, just that it gets executed right when you run it and not at the scheduled time.

    Good question though while zip complains that there is no file. Do you have sudo installed on the server as the zip mode runs as web user and executes zip via sudo to do that.
     
  7. JanThiel

    JanThiel New Member HowtoForge Supporter

    sudo is up and running. I also run the cron_debug using sudo as you can see in the shell output.
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    Using sudo for s cript which needs to be run as root might give strange or wrong results sometimes, better become root user first and then start the script like this:

    sudo -s
    php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php
     
  9. JanThiel

    JanThiel New Member HowtoForge Supporter

    Hi Till,
    finally found some time to dig in again.
    Running the cron again as root it went through. But still - with mode "owned by user as zip" - , there is NO file at /var/www/[DOMAIN]/backup/. The file was created at /var/www/backup/web1/. The interface does not list any backups either.

    Changing the mode to "backup as root" creates a tar.gz file at the same location. But still nothing within the interface.

    Something seems to be very broken here - sadly :-/

    Any more ideas?

    Best Regards,
    Jan
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Try to run an ispconfig update on the slave server and choose to reconfigure permissions in master database during update. When the slave is not able to update the master database, then there might be a permission problem with the ispcsrv* user of this slave server,
     
  11. JanThiel

    JanThiel New Member HowtoForge Supporter

    Done. No problems:
    Code:
     [...] php update.php
    
    
    --------------------------------------------------------------------------------
     _____ ___________   _____              __ _         ____
    |_   _/  ___| ___ \ /  __ \            / _(_)       /__  \
      | | \ `--.| |_/ / | /  \/ ___  _ __ | |_ _  __ _    _/ /
      | |  `--. \  __/  | |    / _ \| '_ \|  _| |/ _` |  |_ |
     _| |_/\__/ / |     | \__/\ (_) | | | | | | | (_| | ___\ \
     \___/\____/\_|      \____/\___/|_| |_|_| |_|\__, | \____/
                                                  __/ |
                                                 |___/
    --------------------------------------------------------------------------------
    
    
    >> Update
    
    Operating System: Centos 7.6
    
    This application will update ISPConfig 3 on your server.
    
    Shall the script create a ISPConfig backup in /var/backup/ now? (yes,no) [yes]: no
    
    Checking ISPConfig database .. OK
    Starting incremental database update.
    Loading SQL patch file: /tmp/ispconfig3_install/install/sql/incremental/upd_dev_collection.sql
    Reconfigure Permissions in master database? (yes,no) [no]: yes
    ...
    Update finished.
    
    Any other ideas or ways to debug this deeper?
     
    Last edited: Jun 14, 2019
  12. JanThiel

    JanThiel New Member HowtoForge Supporter

    @till Any other suggestions how to proceed / debug?
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    You can try to debug the sql connection. Use the 'mysql' command to connect from this slave server to master by using the master database access user and details (the ispcsrv* user, hostname and password from /usr/local/ispconfig/server/lib/config.inc.php file) and then run this mysql query:

    SELECT * FROM web_backup WHERE server_id = 1;

    replace 1 by the server ID of the slave. If this works without mysql error, then you can try an insert operation on that table, something like:

    INSERT INTO web_backup (server_id, parent_domain_id, backup_type, backup_mode, tstamp, filename, filesize) VALUES (1, 2, 'web', 'userzip', 1560784056, 'backup.zip', 99999)

    replace 1 with the server id and 2 with the web id.
     
  14. JanThiel

    JanThiel New Member HowtoForge Supporter

    Hi @till ,
    finally tried out your suggested steps.

    Some more context information:
    - the dbispconfig_[SLAVE_WEB] and dbispconfig_[SLAVE_DB] databases are filled correctly. The web_backup table contains the correctly created backups (as written, the web page backup still only works in root mode).
    - The web_backup table in the dbispconfig (master) DB is empty
    - all databases run on the same db server

    Both SQL commands work as expected. I used the local mysql connection from the slave using the credentials from the config.inc.php.
    The SELECT returned 0 results.
    After the INSERT the dummy record is displayed in the Interface.
    Code:
    MariaDB [dbispconfig]> INSERT INTO web_backup (server_id, parent_domain_id, backup_type, backup_mode, tstamp, filename, filesize) VALUES (2, 1, 'web', 'userzip', 1560784056, 'backup.zip', 99999);
    Query OK, 1 row affected (0.00 sec)
    
    MariaDB [dbispconfig]> SELECT * FROM web_backup WHERE server_id = 2;
    +-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
    | backup_id | server_id | parent_domain_id | backup_type | backup_mode | tstamp     | filename   | filesize |
    +-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
    |         2 |         2 |                1 | web         | userzip     | 1560784056 | backup.zip | 99999    |
    +-----------+-----------+------------------+-------------+-------------+------------+------------+----------+
    1 row in set (0.00 sec)
    
    upload_2019-6-27_15-54-42.png

    Soo ... any other ideas?
     
  15. JanThiel

    JanThiel New Member HowtoForge Supporter

    *bump*
    @till The issue still persist. Should I open up an issue in the tracker for it?
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, please add it to the issue tracker.
     

Share This Page