3 Setting Up The GlusterFS Client
client1.example.com:
On the client, we can install the GlusterFS client as follows:
urpmi glusterfs-client glusterfs-server
Then we create the following directory:
mkdir /mnt/glusterfs
Next we create the file /etc/glusterfs/glusterfs.vol:
vi /etc/glusterfs/glusterfs.vol
volume remote1 type protocol/client option transport-type tcp option remote-host server1.example.com option remote-subvolume brick end-volume volume remote2 type protocol/client option transport-type tcp option remote-host server2.example.com option remote-subvolume brick end-volume volume remote3 type protocol/client option transport-type tcp option remote-host server3.example.com option remote-subvolume brick end-volume volume remote4 type protocol/client option transport-type tcp option remote-host server4.example.com option remote-subvolume brick end-volume volume replicate1 type cluster/replicate subvolumes remote1 remote2 end-volume volume replicate2 type cluster/replicate subvolumes remote3 remote4 end-volume volume distribute type cluster/distribute subvolumes replicate1 replicate2 end-volume volume writebehind type performance/write-behind option window-size 1MB subvolumes distribute end-volume volume cache type performance/io-cache option cache-size 512MB subvolumes writebehind end-volume |
Make sure you use the correct server hostnames or IP addresses in the option remote-host lines!
That's it! Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:
glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs
or
mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs
You should now see the new share in the outputs of...
mount
[root@client1 administrator]# mount
/dev/sda1 on / type ext4 (rw,relatime)
none on /proc type proc (rw)
/dev/sda6 on /home type ext4 (rw,relatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072)
[root@client1 administrator]#
... and...
df -h
[root@client1 administrator]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 12G 1.5G 9.8G 13% /
/dev/sda6 16G 172M 16G 2% /home
/etc/glusterfs/glusterfs.vol
58G 1.7G 56G 3% /mnt/glusterfs
[root@client1 administrator]#
(The size of the distributed storage is calculated by replication1 + replication2, where both replication volumes are as big as the smallest brick.)
Instead of mounting the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted automatically when the client boots.
Open /etc/fstab and append the following line:
vi /etc/fstab
[...] /etc/glusterfs/glusterfs.vol /mnt/glusterfs glusterfs defaults 0 0 |
To test if your modified /etc/fstab is working, reboot the client:
reboot
After the reboot, you should find the share in the outputs of...
df -h
... and...
mount
4 Testing
Now let's create some test files on the GlusterFS share:
client1.example.com:
touch /mnt/glusterfs/test1
touch /mnt/glusterfs/test2
touch /mnt/glusterfs/test3
touch /mnt/glusterfs/test4
touch /mnt/glusterfs/test5
touch /mnt/glusterfs/test6
Now let's check the /data/export directory on server1.example.com, server2.example.com, server3.example.com, and server4.example.com. You will notice that replication1 as well as replication2 hold only a part of the files/directories that make up the GlusterFS share on the client, but the nodes that make up replication1 (server1 and server2) or replication2 (server3 and server4) contain the same files (mirroring):
server1.example.com:
ls -l /data/export
[root@server1 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test5
[root@server1 administrator]#
server2.example.com:
ls -l /data/export
[root@server2 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test5
[root@server2 administrator]#
server3.example.com:
ls -l /data/export
[root@server3 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test6
[root@server3 administrator]#
server4.example.com:
ls -l /data/export
[root@server4 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test6
[root@server4 administrator]#
Now we shut down server1.example.com and server4.example.com and add/delete some files on the GlusterFS share on client1.example.com.
server1.example.com/server4.example.com:
shutdown -h now
client1.example.com:
rm -f /mnt/glusterfs/test5
rm -f /mnt/glusterfs/test6
The changes should be visible in the /data/export directory on server2.example.com and server3.example.com:
server2.example.com:
ls -l /data/export
[root@server2 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
[root@server2 administrator]#
server3.example.com:
ls -l /data/export
[root@server3 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
[root@server3 administrator]#
Let's boot server1.example.com and server4.example.com again and take a look at the /data/export directory:
server1.example.com:
ls -l /data/export
[root@server1 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test5
[root@server1 administrator]#
server4.example.com:
ls -l /data/export
[root@server4 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test6
[root@server4 administrator]#
As you see, server1.example.com and server4.example.com haven't noticed the changes that happened while they were down. This is easy to fix, all we need to do is invoke a read command on the GlusterFS share on client1.example.com, e.g.:
client1.example.com:
ls -l /mnt/glusterfs/
[root@client1 administrator]# ls -l /mnt/glusterfs/
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
[root@client1 administrator]#
Now take a look at the /data/export directory on server1.example.com and server4.example.com again, and you should see that the changes have been replicated to these nodes:
server1.example.com:
ls -l /data/export
[root@server1 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test1
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test2
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test4
[root@server1 administrator]#
server4.example.com:
ls -l /data/export
[root@server4 administrator]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2009-12-21 15:41 test3
[root@server4 administrator]#
5 Links
- GlusterFS: http://www.gluster.org/
- Mandriva: http://www.mandriva.com/