File Serving Cluster - Architecture Questions

[mejalx]

I posted this over at webhostingtalk.com and got nothing other than the usual advertisement spam from random hosting companies. So I decided to ask here :)

Hello guys, I've come for some advice regarding the architecture of a file serving cluster. Let me give you some background information.

I run a gaming community, and we've recently become quite popular. We allow our members to upload gaming related photos, videos, and files.

Currently we're running one dedicated server providing the web app/database, and another cheap unmetered box (lighttpd) for serving the static files. We're literally growing exponentially and I foresee the need to add 2 to 3 file servers within the next month.

What would an ideal architecture be to host/manage these fileservers?

Here is what I've come up with, but somehow I think it's not as good as it should be.

- central application server / db server
- central upload server
- cheap unmetered fileserver boxes

The central upload server will be the file upload gateway. It'll mount each of the cheap unmetered servers through NFS, and then appropriately distribute the files to one of the fileservers.

The problems I see with this are:
1. Load balancing (if a file is taking huge amounts of bandwidth, I'd like to distribute it across multiple servers, but how do I do this when I give a user an absolute link to his or her file like img1.myserver.com/545/file.zip). I suppose I could do this with rewrite rules, but that seems super excessive.
2. NFS - I've read many bad things.
3. Moving files ( same dilemma as load balancing)

If anyone could shed some light or give me some advice, that would be amazing.

Thanks,
Alex

[heymrdj]

Combine http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat with http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster

Mirror with http://www.howtoforge.com/mirroring_with_rsync

Build your central app and DB server. Either get a good box, or use the apache cluster and mysql cluster as I mentioned.

Next make your central upload server and set it up with rsync

Next gather your fileserver boxes and set them all up with RSYNC. Have them use clustered NFS. Have Rsync run every 1-6 hours, however often you feel you need to update content.

Stipulations:
Identical file boxes- servers need to be basically same port speed and same HD size for easy management.
Heartbeat servers: you need more than one hearbeat server so that it's high availability. Either two small root access hosting accounts could be used, or you could set one on the app server and the other on the upload server


Anyways, thats how I would do it. The key is Rsync to keep all server archives absolutely identical and Apache Clustering to make them all load balance. I'm sure you could use Lighthttpd somewhere there instead, but I'm unfamiliar with it (Apache/Mongrel fan here) so that would be your own turf, sorry.

Good luck with your project, sounds fun, wish I was there :).

[heymrdj]

Combine http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat with http://www.howtoforge.com/high_availability_loadbalanced_apache_cluster

Mirror with http://www.howtoforge.com/mirroring_with_rsync

Build your central app and DB server. Either get a good box, or use the apache cluster and mysql cluster as I mentioned.

Next make your central upload server and set it up with rsync

Next gather your fileserver boxes and set them all up with RSYNC. Have them use clustered NFS. Have Rsync run every 1-6 hours, however often you feel you need to update content.

Stipulations:
Identical file boxes- servers need to be basically same port speed and same HD size for easy management.
Heartbeat servers: you need more than one hearbeat server so that it's high availability. Either two small root access hosting accounts could be used, or you could set one on the app server and the other on the upload server


Anyways, thats how I would do it. The key is Rsync to keep all server archives absolutely identical and Apache Clustering to make them all load balance. I'm sure you could use Lighthttpd somewhere there instead, but I'm unfamiliar with it (Apache/Mongrel fan here) so that would be your own turf, sorry.

Good luck with your project, sounds fun, wish I was there :).

Um, was this what you needed or you need me to revise the plan? Work with me here lol :)

[technick]

Sounds to me like ZFS would be a better fit for your distributed file storage.

http://en.wikipedia.org/wiki/ZFS