Website backups apparently still included in quota limits...

Discussion in 'ISPConfig 3 Priority Support' started by invino, Sep 12, 2018.

  1. invino

    invino Member HowtoForge Supporter

    Hi everyone,

    I'm trying to solve a problem and I couldn't find any response, neither in official documentation nor on this forum.

    The problem is this: some websites stop being backed up after their used space reaches a certain limit. Of course, I made sure that the option "Include backups in website quotas" (in System > Interface > Main Config) is unchecked.

    Here is the space used by the website
    Capture d’écran 2018-09-12 à 12.53.33.png

    And here is the backup section of this website
    Capture d’écran 2018-09-12 à 12.53.07.png

    As you can see, the last backup goes back more than a month ago, which is not what the client pays for, obviously.

    My first assumption was that I could find some traces in ISPConfig logs if I switched LogLevel from Errors to Debug. But I couldn't find anything but regular stuff... I have other backups that are configured daily so this was a pretty quick test: I only had to wait for 24hrs if I wanted to find out. But I had no luck in finding anything mentioning quotas, neither in /var/log/ispconfig/ispconfig.log nor /var/log/ispconfig/cron.log.

    Now, something I noticed from the setup that my previous SysAdmin did. Backups are sent to /var/backup in ISPConfig (which is usual) but, on the server, this folder is actually a symlink leading to /home/backup. Could that be the source of my problem?

    Otherwise, what kind of tests could I make to find out where the problem is coming from? I exhausted all ideas I had...

    Thanks for your help!

  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Did you check that the client has not requested to get a copy of his backup in the backup folder of the website?
  3. invino

    invino Member HowtoForge Supporter

    Hi Till,

    Thanks for your really quick response :)

    I just checked for a few of them and nothing is in the backup folder. For any of them...
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    Please take a look at the actual backup files in /home/backup, which Linux user does own them? The websites themselves are in /var/www, right?
  5. invino

    invino Member HowtoForge Supporter

    OK, some folders (e.g. /home/backup/web1) have web users as owners (e.g. client0/web1) and most of the other ones have root/root.
    I'm trying to make a link between users and the issue I'm not sure it's related. It doesn't seem to be normal, though, is it?

    All of the archives are themselves with the right permissions though: root/root
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    When the archives have root:root permissions, then they are not included in the website quota. The differences in folder permissions is probably caused by changes in ispconfig code over the time, so nothing to worry about as the archive owner is what matters for quota.

    Which backup mode did you use under System > server config ?
  7. invino

    invino Member HowtoForge Supporter

    OK, good to know!
    I'm using "Backup web files owned by web user as zip" for the backup mode.
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    This might be part of the problem, the actual backup is created as web user, which means that the zip file is written as web user and later chowned to the root user. So when a single site backup exceeds the remaining quota, then it will probably fail when the location where the data is stored has also quota enabled. Is /home/backup on the same harddisk partition as /var/www? If yes, they share the same backup. If no, does the partition of /home/backup has also user quotas enabled in /etc/fstab?
  9. invino

    invino Member HowtoForge Supporter

    Mmmm... very interesting!

    I have this when I do a df -h:
    Filesystem     Size     Used    Avail   Use%   Mounted on
    /dev/vda5      9.8G     6.6G    2.7G    71%    /var
    /dev/vdb       1007G    482G    475G    51%    /home
    I removed useless lines but there is no mention of /var/www anywhere. So, to me, they are on 2 different partitions, right?

    In /etc/fstab, here is what I have:
    /dev/vda5   /var      ext4    defaults,nosuid                               0    2
    /dev/vdb    /home     ext4    defaults,nosuid,nodev,usrquota,grpquota       0    2
    What do you think?
    Last edited: Sep 12, 2018
  10. invino

    invino Member HowtoForge Supporter

    Sorry, I've just reformated the fstab output.
  11. invino

    invino Member HowtoForge Supporter

    I'm digging more and more into this and I'm remembering something now: we had to bind mount /var/www to /home/www because it was originally a symlink and it was not playing well with ISPConfig (we had lots of errors because of it). Now it's working fine (by the way, we made sure to ignore the duplicated folder that it created in our backups) but here is another issue now, obviously...

    Anyway, if I understand well, usrquota and grpquota are both active on /home, which is used for our websites (/home/www) and backups (/home/backup). Hence the problem with quotas.

    What do you suggest? On my end, I think I just have to mount another disk and move my backups there by changing the path in ISPConfig but maybe you have other ideas :)

  12. till

    till Super Moderator Staff Member ISPConfig Developer

    There is one other thing that you might try first, change the value of the setting 'Backup tmp directory for zip' to a directory which is not on a partition with quota, e.g. use /var/tmp, maybe that helps.

    If not, then you have two options: Either change the backup mode to the one that runs as root user or get another disk / partition to store the backups that does not has quota enabled.
  13. invino

    invino Member HowtoForge Supporter

    Thanks Till!

    I'm already using /tmp and according to fstab, it's not using quotas so I don't think it can help.
    /dev/vda6       /tmp      ext4    defaults,nosuid,nodev      0     2
    If I change ISPConfig backup mode to use the root user, I open the server to some risks according to the documentation. I don't really like this option, to be honest :-/

    OK, so I guess there is this last one solution at the end: mount a new disk on something like S3 for example.

    Am I correct or am I missing something here?
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    I think so, at least when the cause of the issue is what I expect it to be.
  15. invino

    invino Member HowtoForge Supporter

    Thanks Till,

    I will explore this solution then. Simple question though: is this problem coming from the fact our data was moved from /var to /home?
    When I check another install of ISPConfig, I see all data is under /var (backups inclided) with quotas and everything works fine... I'm a little lost here and I'd like to understand, if possible ;-)

    Thanks again!
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    No, that's not related. In my opinion, the reason for the problem is that the partition where the backup storage folder (/home/backup) is located on has quota enabled.

    Maybe you do not have that exact same case on the other server as the backup problem will only occur when a compressed single backup exceeds the remaining quota of the web user, the number of backups don't matter as they are stored as root, but during backup creation, the backup file is written as the user of that web in zip backup mode.
  17. invino

    invino Member HowtoForge Supporter

    If I understand you well, this means we should always store our backups on another partition without quotas, right? Otherwise, there's a conflict. I didn't see this warning anywhere, though...
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, it's safer to do so. And it depends on the backup mode you've chosen.
  19. invino

    invino Member HowtoForge Supporter

    Thanks for the advice! Fair enough for the backup mode, this is actually important to note here, you're totally right ;-)

    OK, so I've got some news. Most of the backups are correctly written to the new disk BUT not the big ones I was actually waiting for! Damn it...

    Good news is I think I found the issue though: our /tmp folder is sized at 1GB, which is precisely the limit over which I don't see backups anymore! I think we just need to resize this /tmp folder properly and we'll be good. And that would (maybe) explain why I couldn't find any errors in the logs...

    Am I right?
  20. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, that can indeed be the case.

Share This Page