Distributed Storage Across Four Storage Nodes With GlusterFS On CentOS 5.4 - Page 2
3 Setting Up The GlusterFS Client
client1.example.com:
GlusterFS isn't available as a package for CentOS 5.4, therefore we have to build it ourselves. First we install the prerequisites:
yum groupinstall 'Development Tools'
yum groupinstall 'Development Libraries'
yum install libibverbs-devel fuse-devel
Then we load the fuse kernel module...
modprobe fuse
... and create the file /etc/rc.modules with the following contents so that the fuse kernel module will be loaded automatically whenever the system boots:
vi /etc/rc.modules
modprobe fuse |
Make the file executable:
chmod +x /etc/rc.modules
Then we download the GlusterFS 2.0.9 sources (please note that this is the same version that is installed on the server!) and build GlusterFS as follows:
cd /tmp
wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.9.tar.gz
tar xvfz glusterfs-2.0.9.tar.gz
cd glusterfs-2.0.9
./configure
At the end of the ./configure command, you should see something like this:
[...]
GlusterFS configure summary
===========================
FUSE client : yes
Infiniband verbs : yes
epoll IO multiplex : yes
Berkeley-DB : yes
libglusterfsclient : yes
argp-standalone : no
make && make install
ldconfig
Check the GlusterFS version afterwards (should be 2.0.9):
glusterfs --version
[root@client1 glusterfs-2.0.9]# glusterfs --version
glusterfs 2.0.9 built on Mar 1 2010 15:58:06
Repository revision: v2.0.9
Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@client1 glusterfs-2.0.9]#
Then we create the following two directories:
mkdir /mnt/glusterfs
mkdir /etc/glusterfs
Next we create the file /etc/glusterfs/glusterfs.vol:
vi /etc/glusterfs/glusterfs.vol
volume remote1 type protocol/client option transport-type tcp option remote-host server1.example.com option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp option remote-host server2.example.com option remote-subvolume brick end-volume volume remote3 type protocol/client option transport-type tcp option remote-host server3.example.com option remote-subvolume brick end-volume volume remote4 type protocol/client option transport-type tcp option remote-host server4.example.com option remote-subvolume brick end-volume volume distribute type cluster/distribute subvolumes remote1 remote2 remote3 remote4 end-volume volume writebehind type performance/write-behind option window-size 1MB subvolumes distribute end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes writebehind end-volume |
Make sure you use the correct server hostnames or IP addresses in the option remote-host lines!
That's it! Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:
glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs
or
mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs
You should now see the new share in the outputs of...
mount
[root@client1 ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
glusterfs#/etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse (rw,allow_other,default_permissions,max_read=131072)
[root@client1 ~]#
... and...
df -h
[root@client1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
29G 2.2G 25G 9% /
/dev/sda1 99M 13M 82M 14% /boot
tmpfs 187M 0 187M 0% /dev/shm
glusterfs#/etc/glusterfs/glusterfs.vol
112G 2.3G 110G 2% /mnt/glusterfs
[root@client1 ~]#
(server1.example.com, server2.example.com, server3.example.com, and server4.example.com each have about 28GB of space for the GlusterFS filesystem, so that the resulting share has a size of about 4 x 28GB (112GB).)
Instead of mounting the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted automatically when the client boots.
Open /etc/fstab and append the following line:
vi /etc/fstab
[...] /etc/glusterfs/glusterfs.vol /mnt/glusterfs glusterfs defaults 0 0 |
To test if your modified /etc/fstab is working, reboot the client:
reboot
After the reboot, you should find the share in the outputs of...
df -h
... and...
mount
4 Testing
Now let's create some test files on the GlusterFS share:
client1.example.com:
touch /mnt/glusterfs/test1
touch /mnt/glusterfs/test2
touch /mnt/glusterfs/test3
touch /mnt/glusterfs/test4
touch /mnt/glusterfs/test5
touch /mnt/glusterfs/test6
Now let's check the /data/export directory on server1.example.com, server2.example.com, server3.example.com, and server4.example.com. You will notice that each storage node holds only a part of the files/directories that make up the GlusterFS share on the client:
server1.example.com:
ls -l /data/export
[root@server1 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test1
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test2
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test5
[root@server1 ~]#
server2.example.com:
ls -l /data/export
[root@server2 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test4
[root@server2 ~]#
server3.example.com:
ls -l /data/export
[root@server3 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test6
[root@server3 ~]#
server4.example.com:
ls -l /data/export
[root@server4 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 20:15 test3
[root@server4 ~]#
5 Links
- GlusterFS: http://www.gluster.org/
- CentOS: http://www.centos.org/
