File Serving Cluster - Architecture Questions

Discussion in 'Server Operation' started by mejalx, Nov 2, 2007.

  1. mejalx

    mejalx New Member

    I posted this over at webhostingtalk.com and got nothing other than the usual advertisement spam from random hosting companies. So I decided to ask here :)

    Hello guys, I've come for some advice regarding the architecture of a file serving cluster. Let me give you some background information.

    I run a gaming community, and we've recently become quite popular. We allow our members to upload gaming related photos, videos, and files.

    Currently we're running one dedicated server providing the web app/database, and another cheap unmetered box (lighttpd) for serving the static files. We're literally growing exponentially and I foresee the need to add 2 to 3 file servers within the next month.

    What would an ideal architecture be to host/manage these fileservers?

    Here is what I've come up with, but somehow I think it's not as good as it should be.

    - central application server / db server
    - central upload server
    - cheap unmetered fileserver boxes

    The central upload server will be the file upload gateway. It'll mount each of the cheap unmetered servers through NFS, and then appropriately distribute the files to one of the fileservers.

    The problems I see with this are:
    1. Load balancing (if a file is taking huge amounts of bandwidth, I'd like to distribute it across multiple servers, but how do I do this when I give a user an absolute link to his or her file like img1.myserver.com/545/file.zip). I suppose I could do this with rewrite rules, but that seems super excessive.
    2. NFS - I've read many bad things.
    3. Moving files ( same dilemma as load balancing)

    If anyone could shed some light or give me some advice, that would be amazing.

    Thanks,
    Alex
     
  2. heymrdj

    heymrdj ISPConfig Developer ISPConfig Developer

    Combine http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat with http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster

    Mirror with http://www.howtoforge.com/mirroring_with_rsync

    Build your central app and DB server. Either get a good box, or use the apache cluster and mysql cluster as I mentioned.

    Next make your central upload server and set it up with rsync

    Next gather your fileserver boxes and set them all up with RSYNC. Have them use clustered NFS. Have Rsync run every 1-6 hours, however often you feel you need to update content.

    Stipulations:
    Identical file boxes- servers need to be basically same port speed and same HD size for easy management.
    Heartbeat servers: you need more than one hearbeat server so that it's high availability. Either two small root access hosting accounts could be used, or you could set one on the app server and the other on the upload server


    Anyways, thats how I would do it. The key is Rsync to keep all server archives absolutely identical and Apache Clustering to make them all load balance. I'm sure you could use Lighthttpd somewhere there instead, but I'm unfamiliar with it (Apache/Mongrel fan here) so that would be your own turf, sorry.

    Good luck with your project, sounds fun, wish I was there :).
     
  3. heymrdj

    heymrdj ISPConfig Developer ISPConfig Developer

    Um, was this what you needed or you need me to revise the plan? Work with me here lol :)
     
  4. technick

    technick New Member

Share This Page