High-Availability Storage Cluster With GlusterFS On Ubuntu

1. Introduction

Original article: http://blogama.org

In this tutorial I will show you how to install GlusterFS in a scalable way to create a storage cluster, starting with 2 servers on Ubuntu 8.04 LTS server. Files will be replicated and splitted accross all servers which is some sort of RAID 10 (raid 1 with < 4 servers). With 4 servers that have each 100GB hard drive, total storage will be 200GB and if one server fails, the data will still be intact and files on the failed server will be replicated on another working server.

GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86-64 server with SATA-II RAID and Infiniband HBA.

 

2. Installation

First you need to install some software:

sudo su

apt-get install sshfs build-essential flex bison byacc vim wget

Now we need to install fuse from source:

cd /root/
wget http://europe.gluster.org/glusterfs/fuse/fuse-2.7.4glfs11.tar.gz
tar -zxvf fuse-2.7.4glfs11.tar.gz
cd /root/fuse-2.7.4glfs11

Next we compile fuse:

./configure
make && make install

Next we will install GlusterFS:

Get the same exact version, otherwise there is good chances it wont work. I tried with 2.0.0rc1 and 1.3.12 and there was some issues (1.4.0rc7 works fine).

cd /root/
wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.0rc2.tar.gz
tar -zxvf glusterfs-2.0.0rc2.tar.gz
cd /root/glusterfs-2.0.0rc2/

Take a minute break and compile:

./configure
make && make install

For some reasons, libraries are going in the wrong directory so we need to (if someone has a clean fix to this please post it!):

cp /usr/local/lib/* -R /usr/lib/

Next we create some folders that will be used later on:

mkdir /mnt/glusterfs
mkdir /data/
mkdir /data/export
mkdir /data/export-ns
mkdir /etc/glusterfs/

 

3. Servers configuration

Before you go further, you need to know that GlusterFS works in a client/server way. What we will do is to make our servers both client and server for GlusterFS.

Lets start with the server configuration file ON ALL SERVERS:

vi /etc/glusterfs/glusterfs-server.vol

and make it look like this:

# file: /etc/glusterfs/glusterfs-server.vol
 
volume posix
  type storage/posix
  option directory /data/export
end-volume
 
volume locks
  type features/locks
  subvolumes posix
end-volume
 
volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume
 
volume posix-ns
  type storage/posix
  option directory /data/export-ns
end-volume
 
volume locks-ns
  type features/locks
  subvolumes posix-ns
end-volume
 
volume brick-ns
  type performance/io-threads
  option thread-count 8
  subvolumes locks-ns
end-volume
 
volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.brick.allow *
  option auth.addr.brick-ns.allow *
  subvolumes brick brick-ns
end-volume

Now do:

glusterfsd -f /etc/glusterfs/glusterfs-server.vol

to start the server daemon.

Share this page:

9 Comment(s)

Add comment

Comments

From: Mike Glass at: 2009-04-09 19:26:16

For whatever reason my system would only

When configuring with --prefix= I got the error:

libtool: install: error: cannot install `libglusterfsclient.la' to a directory not ending in /usr/lib

 to fix this I did this:

./configure --prefix=/usr --sysconfdir=/etc --datadir=/var

 This might be a slightly cleaner solution then the aforementioned one if you want to keep your files in order.

From: Anonymous at: 2009-08-07 00:49:50

You have to regenerate configure by running ./autogen first.

From: Anne Oni Mouse at: 2009-03-04 22:50:39

For some reasons, libraries are going in the wrong directory so we need to (if someone has a clean fix to this please post it!) ... That's ./configure, you have to tell it where to put libraries... try --prefix=

From: Anonymous at: 2009-03-06 03:22:29

This was done for local storage on a multi node webserver. Im sure with heartbeat you can easily do that.

 

Marc

From: at: 2009-11-11 14:33:50

Hi to everybody.

I've followed exatly this howto and I've realized a RAID1 between 2 servers.

The problem is that is working just if SRV1 is always up and SRV2 can reboot or crash without problem.

If, for some reasons, SVR2 will be rebooted, I still have the replication of SRV1, but I can not write  in the replicated folder (I've this error: Transport endpoint is not connected)

What is wrong with this configuration? I've this two files in both servers:

SERVER:

# file: /etc/glusterfs/glusterfsd.vol
volume posix
type storage/posix
option directory /data/export
end-volume

volume locks
type features/locks
subvolumes posix
end-volume

volume brick
type performance/io-threads
option thread-count 8
subvolumes locks
end-volume

volume posix-ns
type storage/posix
option directory /data/export-ns
end-volume

volume locks-ns
type features/locks
subvolumes posix-ns
end-volume

volume brick-ns
type performance/io-threads
option thread-count 8
subvolumes locks-ns
end-volume

volume server
type protocol/server
option transport-type tcp
option auth.addr.brick.allow *
option auth.addr.brick-ns.allow *
subvolumes brick brick-ns
end-volume

 CLIENT:

# file: /etc/glusterfs/glusterfs.vol

### Add client feature and attach to remote subvolume of server1

volume brick1
type protocol/client
option transport-type tcp/client
option remote-host 192.168.0.1 # IP address of the remote brick
option remote-subvolume brick # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server2
volume brick2
type protocol/client
option transport-type tcp/client
option remote-host 192.168.0.2 # IP address of the remote brick
option remote-subvolume brick # name of the remote volume
end-volume

### The file index on server1
volume brick1-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.0.1 # IP address of the remote brick
option remote-subvolume brick-ns # name of the remote volume
end-volume

### The file index on server2
volume brick2-ns
type protocol/client
option transport-type tcp/client
option remote-host 192.168.0.2 # IP address of the remote brick
option remote-subvolume brick-ns # name of the remote volume
end-volume

#The replicated volume with data
volume afr1
type cluster/afr
subvolumes brick1 brick2
end-volume

#The replicated volume with indexes
volume afr-ns
type cluster/afr
subvolumes brick1-ns brick2-ns
end-volume

#The unification of all afr volumes (used for > 2 servers)
volume unify
type cluster/unify
option scheduler rr # round robin
option namespace afr-ns
subvolumes afr1
end-volume

From: john at: 2010-04-11 23:47:30

Could this be the reason it isn't self-healing?...

http://www.gluster.com/community/documentation/index.php/Whats_New_v2.0#Known_Issues

Specifically: "Replicate will only self-heal if the files exist on the first subvolume. Server A-> B works, Server A <-B does not work."

 

From: Nathan at: 2012-01-09 02:25:41

Your article outlines how to set up a redundant filesystem. It doesn't tackle high availability for the filesystem. Please include this information or change the title of the article.

From: katz at: 2009-03-04 17:24:10

I have one doubt.

Do you have any kind of availability example using virtual Ip address between two servers ? Like VRRP or HSRP.

 

From: Anonymous at: 2009-06-19 11:51:36

hi, is this also posibel? It works on my server but i wonder if this is a bad ide or not ?:

#client config

### Add client feature and attach to remote subvolume of server1
volume brick1
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.21    # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
### Add client feature and attach to remote subvolume of server2
volume brick2
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.22      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
### Add client feature and attach to remote subvolume of server3
volume brick3
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.23      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
### Add client feature and attach to remote subvolume of server4
volume brick4
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.24      # IP address of the remote brick
 option remote-subvolume brick        # name of the remote volume
end-volume
 
### Add client feature and attach to remote subvolume of server1
volume brick1-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.21    # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume
 
### Add client feature and attach to remote subvolume of server2
volume brick2-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.22      # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server2
volume brick3-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.23      # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of server2
volume brick4-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host 172.16.2.24      # IP address of the remote brick
 option remote-subvolume brick-ns        # name of the remote volume
end-volume
 
volume afr1
 type cluster/afr
 subvolumes brick1 brick2 brick3 brick4
end-volume

volume afr-ns
 type cluster/afr
 subvolumes brick1-ns brick2-ns brick3-ns brick4-ns
end-volume

volume unify
  type cluster/unify
  option scheduler rr # round robin
  option namespace afr-ns
  subvolumes afr1
end-volume