Distributed Replicated Storage Across Four Storage Nodes With GlusterFS On Debian Lenny - Page 2

This tutorial exists for these OS versions

    On this page

    1. 3 Setting Up The GlusterFS Client
    2. 4 Testing
    3. 5 Links

    3 Setting Up The GlusterFS Client

    client1.example.com:

    On the client, we need to install fuse and GlusterFS. Instead of installing the libfuse2 package from the Debian repository, we install a patched version with better support for GlusterFS.

    First we install the prerequisites again:

    aptitude install sshfs build-essential flex bison byacc libdb4.6 libdb4.6-dev

    Then we build fuse as follows (you can find the latest patched fuse version on ftp://ftp.zresearch.com/pub/gluster/glusterfs/fuse/):

    cd /tmp
    wget ftp://ftp.zresearch.com/pub/gluster/glusterfs/fuse/fuse-2.7.4glfs11.tar.gz
    tar -zxvf fuse-2.7.4glfs11.tar.gz
    cd fuse-2.7.4glfs11
    ./configure
    make && make install

    Afterwards we build GlusterFS (just like on the server)...

    cd /tmp
    wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.1.tar.gz
    tar xvfz glusterfs-2.0.1.tar.gz
    cd glusterfs-2.0.1
    ./configure --prefix=/usr > /dev/null

    make && make install
    ldconfig
    glusterfs --version

    ... and create the following two directories:

    mkdir /mnt/glusterfs
    mkdir /etc/glusterfs

    Next we create the file /etc/glusterfs/glusterfs.vol:

    vi /etc/glusterfs/glusterfs.vol

    volume remote1
      type protocol/client
      option transport-type tcp
      option remote-host server1.example.com
      option remote-subvolume brick
    end-volume
    
    volume remote2
      type protocol/client
      option transport-type tcp
      option remote-host server2.example.com
      option remote-subvolume brick
    end-volume
    
    volume remote3
      type protocol/client
      option transport-type tcp
      option remote-host server3.example.com
      option remote-subvolume brick
    end-volume
    
    volume remote4
      type protocol/client
      option transport-type tcp
      option remote-host server4.example.com
      option remote-subvolume brick
    end-volume
    
    volume replicate1
      type cluster/replicate
      subvolumes remote1 remote2
    end-volume
    
    volume replicate2
      type cluster/replicate
      subvolumes remote3 remote4
    end-volume
    
    volume distribute
      type cluster/distribute
      subvolumes replicate1 replicate2
    end-volume
    
    volume writebehind
      type performance/write-behind
      option window-size 1MB
      subvolumes distribute
    end-volume
    
    volume cache
      type performance/io-cache
      option cache-size 512MB
      subvolumes writebehind
    end-volume

    Make sure you use the correct server hostnames or IP addresses in the option remote-host lines!

    That's it! Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:

    glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs

    or

    mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs

    You should now see the new share in the outputs of...

    mount

    client1:/tmp/glusterfs-2.0.1# mount
    /dev/sda1 on / type ext3 (rw,errors=remount-ro)
    tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
    proc on /proc type proc (rw,noexec,nosuid,nodev)
    sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
    udev on /dev type tmpfs (rw,mode=0755)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
    devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
    fusectl on /sys/fs/fuse/connections type fusectl (rw)
    /etc/glusterfs/glusterfs.vol on /mnt/glusterfs type fuse.glusterfs (rw,max_read=131072,allow_other,default_permissions)
    client1:/tmp/glusterfs-2.0.1#

    ... and...

    df -h

    client1:/tmp/glusterfs-2.0.1# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              29G  935M   27G   4% /
    tmpfs                 126M     0  126M   0% /lib/init/rw
    udev                   10M   80K   10M   1% /dev
    tmpfs                 126M     0  126M   0% /dev/shm
    /etc/glusterfs/glusterfs.vol
                           48G  1.7G   44G   4% /mnt/glusterfs
    client1:/tmp/glusterfs-2.0.1#

    (The size of the distributed storage is calculated by replication1 + replication2, where both replication volumes are as big as the smallest brick.)

    Instead of mounting the GlusterFS share manually on the client, you could modify /etc/fstab so that the share gets mounted automatically when the client boots.

    Open /etc/fstab and append the following line:

    vi /etc/fstab

    [...]
    /etc/glusterfs/glusterfs.vol  /mnt/glusterfs  glusterfs  defaults  0  0

    To test if your modified /etc/fstab is working, reboot the client:

    reboot

    After the reboot, you should find the share in the outputs of...

    df -h

    ... and...

    mount

     

    4 Testing

    Now let's create some test files on the GlusterFS share:

    client1.example.com:

    touch /mnt/glusterfs/test1
    touch /mnt/glusterfs/test2
    touch /mnt/glusterfs/test3
    touch /mnt/glusterfs/test4
    touch /mnt/glusterfs/test5
    touch /mnt/glusterfs/test6

    Now let's check the /data/export directory on server1.example.com, server2.example.com, server3.example.com, and server4.example.com. You will notice that replication1 as well as replication2 hold only a part of the files/directories that make up the GlusterFS share on the client, but the nodes that make up replication1 (server1 and server2) or replication2 (server3 and server4) contain the same files (mirroring):

    server1.example.com:

    ls -l /data/export

    server1:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test5
    server1:/tmp/glusterfs-2.0.1#

    server2.example.com:

    ls -l /data/export

    server2:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test5
    server2:/tmp/glusterfs-2.0.1#

    server3.example.com:

    ls -l /data/export

    server3:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test6
    server3:/tmp/glusterfs-2.0.1#

    server4.example.com:

    ls -l /data/export

    server4:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test6
    server4:/tmp/glusterfs-2.0.1#

    Now we shut down server1.example.com and server4.example.com and add/delete some files on the GlusterFS share on client1.example.com.

    server1.example.com/server4.example.com:

    shutdown -h now

    client1.example.com:

    rm -f /mnt/glusterfs/test5
    rm -f /mnt/glusterfs/test6

    The changes should be visible in the /data/export directory on server2.example.com and server3.example.com:

    server2.example.com:

    ls -l /data/export

    server2:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    server2:/tmp/glusterfs-2.0.1#

    server3.example.com:

    ls -l /data/export

    server3:/tmp/glusterfs-2.0.1# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    server3:/tmp/glusterfs-2.0.1#

    Let's boot server1.example.com and server4.example.com again and take a look at the /data/export directory:

    server1.example.com:

    ls -l /data/export

    server1:~# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test5
    server1:~#

    server4.example.com:

    ls -l /data/export

    server4:~# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test6
    server4:~#

    As you see, server1.example.com and server4.example.com haven't noticed the changes that happened while they were down. This is easy to fix, all we need to do is invoke a read command on the GlusterFS share on client1.example.com, e.g.:

    client1.example.com:

    ls -l /mnt/glusterfs/

    client1:~# ls -l /mnt/glusterfs/
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    client1:~#

    Now take a look at the /data/export directory on server1.example.com and server4.example.com again, and you should see that the changes have been replicated to these nodes:

    server1.example.com:

    ls -l /data/export

    server1:~# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test1
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test2
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test4
    server1:~#

    server4.example.com:

    ls -l /data/export

    server4:~# ls -l /data/export
    total 0
    -rw-r--r-- 1 root root 0 2009-06-03 15:24 test3
    server4:~#

     

    Share this page:

    0 Comment(s)

    Add comment