S3 bucket as backup folder

Discussion in 'Installation/Configuration' started by progressed36, Apr 18, 2020.

  1. progressed36

    progressed36 Member

    Hi there,

    Im thinking about mounting a S3 bucket as folder to move the backup files to another location and away from the same physical location as the server itself. Im hoping someone has done this already and could guide me into choosing the right approach.

    1. mount the bucket with riofs and point the backupfolder to that mountpoint, but will this work without hassle seeing as it has to check for other files and delete older than the settings made in retention etc.
    2. keep the current backupfolder and mv the files to that point every day and manually delete every month some files.
    3. other option?
     
  2. nhybgtvfr

    nhybgtvfr Active Member

    I've done it before using s3fs. I just mounted the s3 bucket to /var/backup and let it write the backups straight to s3.
    works fine on a single server. tried it with multi-server, changing the backup paths on each server to be /var/backup/[SERVERNAME]/
    and still mounted the s3 bucket to /var/backup/ it didn't particularly like multiple servers writing to the same bucket at the same time, even though there was no attempt to write to the same folder/file from different servers, so make sure it's only 1 server at a time.

    after that I tried objectiveFS, a shared posix file system for s3, you need to pay for it, but it allows for multiple servers to use the same s3 bucket and concurrent write.
     
  3. progressed36

    progressed36 Member

    thanks for your insights. Personally i tried s3fs but immediately abandoned the project after 60 minutes of testing. It was slow and buggy writing the smaller files as it wasn't keeping up. (i know s3 isn't meant for small files). riofs was fast and no errors. objectiveFS looks awesome but im in no need of 5Tb of files yet. Ill test the S3 of Digital ocean (250Gb for 5 dollar / month) and ill report back here.
     
  4. nhybgtvfr

    nhybgtvfr Active Member

    never heard of riofs before, only use objectivefs because a client we support was already using it on their system, so I knew it allowed multiple servers to access a single s3 bucket without problems, also has a lot of options for speeding up or throttling data transfer and caching.
    might have to play around with riofs, see just what it can do, it says it's highly concurrent, but that may just be for multiple processes trying to read/write the same file, will be interesting to see how it handles multiple servers.
     
  5. progressed36

    progressed36 Member

    I will let you know... im testing with goofys now (instead of riofs) starting today. if successful i will add other servers tomorrow (or day after).
    I created a folder on the bucket and mounted directly to that subfolder.

    benchmark details on github vs 3sfs:
    https://github.com/kahing/goofys
     

Share This Page