How to Install and Configure GlusterFS on Ubuntu

GlusterFS is a free, open source and scalable network file system developed first by Gluster Inc. and later by Redhat. It is specifically designed for data-intensive workloads such as cloud storage, CDN and media streaming. It is capable of handling petabytes of data and can therefore serve thousands of clients. It supports various protocols such as NFS, SMB, CIFS, HTTP and FTP. With GlusterFS, you can combine different disk storage resources into a single global namespace.

In this tutorial we will learn how to set up a replicated GlusterFS volume on Ubuntu 18.04.

Requirements

• Two server running Ubuntu 18.04 with 2 GB external HDD on each.
• A static IP address 172.20.10.5 is configured on node1 and 172.20.10.6 is configured on node2.
• A root password is configured on both server.

Getting Started

Before starting, you will need to update both servers with the latest version. You can update them by running the following command on both server.

apt-get update -y
apt-get upgrade -y

Once both servers are updated restart them to apply the changes.

Next, you will need to setup hostname resolution on both servers so each server can communicate with each other by hostname. You can do it by editing /etc/hosts file on both server:

nano /etc/hosts

Add the following lines:

172.20.10.5 node1
172.20.10.6 node2

Save and close the file when you are finished.

Install GlusterFS

First, you will need to install the GlusterFS package on both servers. By default, GlusterFS is not available in the Ubuntu 18.04 default repository. So you will need to add the GlusterFS repository on both servers.

First, install the required package with the following command:

apt-get install software-properties-common -y

Next, add the repository with the following command:

add-apt-repository ppa:gluster/glusterfs-5

Next, install GlusterFS server with the following command:

apt-get install glusterfs-server -y

Once the installation has been completed, start the GlusterFS service and enable it to start on boot with the following command:

systemctl start glusterd
systemctl enable glusterd

You can also check the status of GlusterFS service with the following command:

systemctl status glusterd

You should see the following output:

? glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-11-07 12:51:15 UTC; 48s ago
 Main PID: 7199 (glusterd)
    Tasks: 8 (limit: 505)
   CGroup: /system.slice/glusterd.service
           ??7199 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Nov 07 12:51:06 ubuntu systemd[1]: Starting GlusterFS, a clustered file-system server...
Nov 07 12:51:15 ubuntu systemd[1]: Started GlusterFS, a clustered file-system server.

Create GlusterFS Storage

Next, you will need to create a partition on external harddisk(/dev/sdb) on both server.

To create a new partition on /dev/sdb, run the following command:

fdisk /dev/sdb

You should see the following output:

Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x06091fe8.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-4194303, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-4194303, default 4194303): 

Created a new partition 1 of type 'Linux' and of size 2 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Next, format the partition with the following command:

mkfs.xfs /dev/sdb1

You should see the following output:

meta-data=/dev/sdb1              isize=512    agcount=4, agsize=131008 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=524032, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Next, create a directory for GlusterFS storage on both server with the following command:

mkdir /glusterdata

Next, mount the disk /dev/sdb1 on a directory /glusterdata.

mount.xfs /dev/sdb1 /glusterdata

Next, make the mount persistent across reboot by editing /etc/fstab file:

nano /etc/fstab

Add the following line:

/dev/sdb1 /glusterdata xfs defaults 0 0

Save and close the file then run the following command to apply the changes:

mount -a

You can now check the mounted directory with the following command:

df -h

You should see the following output:

Filesystem      Size  Used Avail Use% Mounted on
udev            211M     0  211M   0% /dev
tmpfs            49M 1016K   48M   3% /run
/dev/sda2        30G  4.3G   25G  16% /
tmpfs           241M   12K  241M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           241M     0  241M   0% /sys/fs/cgroup
/dev/loop0       87M   87M     0 100% /snap/core/4486
tmpfs            49M     0   49M   0% /run/user/0
/dev/loop1       90M   90M     0 100% /snap/core/7917
/dev/sdb1       2.0G   35M  2.0G   2% /glusterdata

Configure GlusterFS Volume

First, you will need to create a trusted storage pool by adding node2 on node1. To do so, run the following command on node1:

gluster peer probe node2

You can list the storage pool with the following command:

gluster pool list

You should get the following output:

UUID					Hostname 	State
d049b519-9210-4bfa-a1c6-0d1454a88ea0	node2    	Connected 
dbb9252e-6082-4d31-aaef-1b99e0e0c369	localhost	Connected 

Next, create a brick directory on both nodes with the following command:

mkdir /glusterdata/vol1

Next, create a volume named vol1 with two replicas:

gluster volume create vol1 replica 2 node1:/glusterdata/vol1 node2:/glusterdata/vol1

Next, start the volume using the following command:

gluster volume start vol1

You can check the status of the created volume with the following command:

gluster volume status

You should get the following output:

Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/glusterdata/vol1               49152     0          Y       7987 
Brick node2:/glusterdata/vol1               49152     0          Y       18565
Self-heal Daemon on localhost               N/A       N/A        Y       8010 
Self-heal Daemon on node2                   N/A       N/A        Y       18588
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

You can also see the information about created volume with the following command:

gluster volume info vol1

You should get the following output:

Volume Name: vol1
Type: Replicate
Volume ID: e093ae7a-70ac-42b8-ab3f-a8bd186c5740
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/glusterdata/vol1
Brick2: node2:/glusterdata/vol1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

Next, mount the created volume on /mnt directory on both nodes by editing /etc/fstab file so your volume to be available after the server boots.

nano /etc/fstab

Add the following line:

localhost:/vol1 /mnt glusterfs defaults,_netdev 0 0

Save and close the file when you are finished. Then, run the following command to apply the changes:

mount -a

Test Replication

GlusterFS server is now installed and configured. It's time to test replication between two nodes.

First, create two files on node1:

touch /mnt/file1 /mnt/file2

Now, go to the node2 and run the following command:

ls -l /mnt/

You should see the same files that you have created on node1:

total 0
-rw-r--r-- 1 root root 0 Nov  7 13:42 file1
-rw-r--r-- 1 root root 0 Nov  7 13:42 file2

Now, stop the GlusterFS service on node2 with the following command:

service glusterd stop

Next, go to the node1 and check the peer status:

gluster peer status

You should see that node2 has been disconnected:

Number of Peers: 1

Hostname: node2
Uuid: d049b519-9210-4bfa-a1c6-0d1454a88ea0
State: Peer in Cluster (Disconnected)

You can also check the volume status with the following command:

gluster volume status

You should see the following output:

Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/glusterdata/vol1               49152     0          Y       7987 
Self-heal Daemon on localhost               N/A       N/A        Y       8010 
 
Task Status of Volume vol1
------------------------------------------------------------------------------
There are no active volume tasks

Next, create two other files on node1:

touch /mnt/file3 /mnt/file4

Now, go to the node2 and start the GlusterFS service with the following command:

service glusterd start

Now, check the /mnt directory on node2:

ls -l /mnt/

You should see that all files from node1 have been replicated and are now available.

total 0
-rw-r--r-- 1 root root 0 Nov  7 13:42 file1
-rw-r--r-- 1 root root 0 Nov  7 13:42 file2
-rw-r--r-- 1 root root 0 Nov  7 13:44 file3
-rw-r--r-- 1 root root 0 Nov  7 13:44 file4

Conclusion

Congratulations! you have successfully installed and configured two-node Replicated Glusterfs Volume on Ubuntu 18.04 server. You can also setup Distributed Glusterfs Volume, Distributed Replicated Glusterfs Volume, Striped Glusterfs Volume and Distributed Striped Glusterfs Volume.