Striping Across Four Storage Nodes With GlusterFS On Ubuntu 9.10 - Page 2

3 Setting Up The GlusterFS Client

client1.example.com:

On the client, we can install the GlusterFS client as follows:

aptitude install glusterfs-client glusterfs-server

Then we create the following directory:

mkdir /mnt/glusterfs

Next we create the file /etc/glusterfs/glusterfs.vol (we make a backup of the original /etc/glusterfs/glusterfs.vol file first):

cp /etc/glusterfs/glusterfs.vol /etc/glusterfs/glusterfs.vol_orig
cat /dev/null > /etc/glusterfs/glusterfs.vol
vi /etc/glusterfs/glusterfs.vol

volume remote1
  type protocol/client
  option transport-type tcp/client
  option remote-host server1.example.com
  option remote-subvolume brick
end-volume

volume remote2
  type protocol/client
  option transport-type tcp/client
  option remote-host server2.example.com
  option remote-subvolume brick
end-volume

volume remote3
  type protocol/client
  option transport-type tcp/client
  option remote-host server3.example.com
  option remote-subvolume brick
end-volume

volume remote4
  type protocol/client
  option transport-type tcp/client
  option remote-host server4.example.com
  option remote-subvolume brick
end-volume

volume stripe
  type cluster/stripe
  option block-size 1MB
  subvolumes remote1 remote2 remote3 remote4
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes stripe
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume

Make sure you use the correct server hostnames or IP addresses in the option remote-host lines!

That's it! Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:

glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs

or

mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs

You should now see the new share in the outputs of...

mount

root@client1:~# mount
/dev/mapper/client1-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/dev/sda5 on /boot type ext2 (rw)
/etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse.glusterfs (rw,max_read=131072,allow_other,default_permissions)
root@client1:~#

... and...

df -h

root@client1:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/client1-root
                       29G  808M   27G   3% /
udev                  122M  152K  121M   1% /dev
none                  122M     0  122M   0% /dev/shm
none                  122M   36K  122M   1% /var/run
none                  122M     0  122M   0% /var/lock
none                  122M     0  122M   0% /lib/init/rw
/dev/sda5             228M   15M  202M   7% /boot
/etc/glusterfs/glusterfs.vol
                      103G  3.2G   95G   4% /mnt/glusterfs
root@client1:~#

(server1.example.com, server2.example.com, server3.example.com, and server4.example.com each have about 26GB of space for the GlusterFS filesystem, so that the resulting share has a size of about 4 x 26GB (103GB).)

Instead of mounting the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted automatically when the client boots.

Open /etc/fstab and append the following line:

vi /etc/fstab

[...]
/etc/glusterfs/glusterfs.vol  /mnt/glusterfs  glusterfs  defaults  0  0

To test if your modified /etc/fstab is working, reboot the client:

reboot

After the reboot, you should find the share in the outputs of...

df -h

... and...

mount

4 Testing

Now let's create a big test file on the GlusterFS share:

client1.example.com:

dd if=/dev/zero of=/mnt/glusterfs/test.img bs=1024k count=1000

ls -l /mnt/glusterfs

root@client1:~# ls -l /mnt/glusterfs
total 1024032
-rw-r--r-- 1 root root 1048576000 2009-12-22 17:31 test.img
root@client1:~#

Now let's check the /data/export directory on server1.example.com, server2.example.com, server3.example.com, and server4.example.com. You should see the test.img file on each node, but with different sizes (due to data striping):

server1.example.com:

ls -l /data/export

root@server1:~# ls -l /data/export
total 256008
-rw-r--r-- 1 root root 1045430272 2009-12-22 17:31 test.img
root@server1:~#

server2.example.com:

ls -l /data/export

root@server2:~# ls -l /data/export
total 256008
-rw-r--r-- 1 root root 1046478848 2009-12-22 17:27 test.img
root@server2:~#

server3.example.com:

ls -l /data/export

root@server3:~# ls -l /data/export
total 256008
-rw-r--r-- 1 root root 1047527424 2009-12-22 17:26 test.img
root@server3:~#

server4.example.com:

ls -l /data/export

root@server4:~# ls -l /data/export
total 256008
-rw-r--r-- 1 root root 1048576000 2009-12-22 17:30 test.img
root@server4:~#

5 Links

• GlusterFS: http://www.gluster.org/
• Ubuntu: http://www.ubuntu.com/