Openfiler 2.3 Active/Passive Cluster (Heartbeat, DRBD) With Offsite Replication Node

Introduction

Openfiler is a Linux based NAS/SAN application which can deliver storage over nfs/smb/iscsi and ftp. It has a web interface over that you can control these services. The howto is based on the Howto from Kyle Gililland. A lot of thanks to him for this.

The cluster we build will consist of two nodes replicating each other and taking over services and storage in case of emergency. Furthermore we have an Offsite Replication Server, which ideally stands in a physically different position and replicates the configurations/storage from which ever node is active. In case of emergency this Offsite Replication Server can be used to restore the cluster and to deliver the services.

I used the Openfiler VMware Images 2.3 x86 and Openfiler 2.3 x86 Installation Media to test out it this howto. Hopefully you will be able to repeat my steps and have a safe and cheap Storage Solution.

 

Overview

  • 1. Installation
  • 2. Prepaire the partitions for DRBD
  • 3. Create basic heartbeat configuration
  • 4. DRBD Configuration
  • 5. Prepaire the configuration partition
  • 6. Initiate Data/Storage partition
  • 7. Create final heartbeat configuration
  • 8. Enable/Disable sytem services
  • 9. Test failover on node1 and node2
  • 10. Test recover of filer01 and filer02
    • 10.1 DRBD Configuration
    • 10.2 filer01 and filer02 redo configuration
    • 10.3 Retake resources and run cluster again
  • 11. Use Replication Node as Main Node
    • 11.1 Finished Replication, how to turn Replication Node in standby again
  • 12. Add another Storage Partition

 

Requirements

Hardware

  • 3x boxes that meet the minimum hardware requirements of openfiler
  • 2x ethernet interfaces at each box

1. Installation

In the Installation process of openfiler leave the portion of the disk you wanna use for Storage Services unused. In this case it was a second harddisk in each system. We create the partitions for Storage Replication after the installation.

Networkconfiguration

  • on filer01:
    • eth0: 10.10.11.101
    • eth1: 10.10.50.101
  • on filer02:
    • eth0: 10.10.11.102
    • eth1: 10.10.50.102
  • on filer03:
    • eth0: 10.10.11.103
    • eth1: 10.10.50.103
  • virtual devices: ( this will be created by heartbeat, dont add these ip's to any interface )
    • eth0:0 10.10.11.100 ( iscsi target server ip )
    • eth1:0 10.10.50.100 ( replication interface )

After finishing the installation build yourself a nice working environment while you update the openfiler systems. It's important to update the systems now as openfiler 2.3 standard installation media dont bring DRBD 8.3.x which is needed for 3-node replication.

conary update conary
conary updateall --replace-files --no-conflict-check

While the systems update you add the other filers to the /etc/hosts file and generate some ssh-keys and exchange them.

[email protected] ~# nano /etc/hosts
#Installed by rBuilder
127.0.0.1       filer01 localhost.localdomain   localhost
10.10.50.102	filer02
10.10.50.103	filer03
[email protected] ~# nano /etc/hosts
#Installed by rBuilder
127.0.0.1       filer02 localhost.localdomain   localhost
10.10.50.101	filer01
10.10.50.103	filer03
[email protected] ~# nano /etc/hosts
#Installed by rBuilder
127.0.0.1       filer03 localhost.localdomain   localhost
10.10.50.101	filer01
10.10.50.102	filer02

Then we create ssh-keys without a passphrase which we use for easier file-exchange.

[email protected] ~# ssh-keygen -t dsa

Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
4c:76:ec:61:10:2a:07:9f:35:8d:9b:16:89:5c:bc:b9 [email protected]

Do the same on the other 2 nodes:

[email protected] ~# ssh-keygen -t dsa
[email protected] ~# ssh-keygen -t dsa

Exchange the public keys between the nodes:

[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys2
[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys2

[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys2
[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys

[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys
[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys

[email protected] ~# cat .ssh/authorized_keys >> .ssh/authorized_keys2
[email protected] ~# cat .ssh/authorized_keys >> .ssh/authorized_keys2
[email protected] ~# cat .ssh/authorized_keys >> .ssh/authorized_keys2

 

2. Prepare The Partitions For DRBD

Setup the partitions for DRBD which will contain the configuration files for openfiler and its storage services. Don't add any of this to /etc/fstab. If you created the partitions in the installation process remove the entries for them at the /etc/fstab file.

You have todo this steps on all 3 Machines.

fdisk /dev/sdb

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1566, default 1566): 67
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (68-1566, default 68):
Using default value 68
Last cylinder, +cylinders or +size{K,M,G} (68-1566, default 1566):
Using default value 1566
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)

You have to exist fdisk with w to write the changed to the disk.

You can check the partitions after this:

fdisk -l

Disk /dev/sda: 2355 MB, 2355978240 bytes
16 heads, 63 sectors/track, 4565 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sda1 * 1 4565 2300696 83 Linux

Disk /dev/sdb: 12.9 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xff9e0345

Device Boot Start End Blocks Id System
/dev/sdb1 1 67 538146 83 Linux
/dev/sdb2 68 1566 12040717+ 8e Linux LVM

NOTE: Before you can continue the systems should have finished update.

 

3. Create A Basic Heartbeat Configuration

This step is needed to create the virtual IP that DRBD is going to use for the offsite replication.

Create /etc/ha.d/authkeys ( filer01 and filer02 ):

auth 2
2 crc

Change the accessrights to the /etc/ha.d/authkeys file:

chmod 600 /etc/ha.d/authkeys

Create etc/ha.d/ha.cf ( filer01 and filer02 ):

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
bcast eth1
keepalive 5
warntime 10
deadtime 120
initdead 120
udpport 694
auto_failback off
node filer01
node filer02

Edit /opt/openfiler/etc/cluster.xml ( filer01 ):

<?xml version="1.0" ?>
<cluster>
<clustering state="on" />
<nodename value="filer01" />
<resource value="MailTo::[email protected]::ClusterFailover"/>
<resource value="IPaddr::10.10.50.100/24/eth1" />
</cluster>

Go to https://10.10.11.101:446 and start iscsi target service ( this will create /etc/ha.d/haresources on filer01 ).

Copy haresource file to filer02 and start heartbeat IP:

[email protected] ~# scp /etc/ha.d/haresources [email protected]:/etc/ha.d/haresources
[email protected] ~# service heartbeat start
[email protected] ~# service heartbeat start

Wait till heartbeat is up on both nodes, then stop heartbeat on filer02:

[email protected] ~# service heartbeat stop

This should create a failover where filer01 gets the ip address 10.11.50.100, check this with ifconfig on filer01, the output should be like this.

[email protected] ~# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:D9:66:27
inet addr:10.10.11.101 Bcast:10.10.11.255 Mask:255.255.255.0


eth1 Link encap:Ethernet HWaddr 00:0C:29:D9:66:31
inet addr:10.10.50.101 Bcast:10.10.50.255 Mask:255.255.255.0


eth1:0 Link encap:Ethernet HWaddr 00:0C:29:D9:66:31
inet addr:10.10.50.100 Bcast:10.10.50.255 Mask:255.255.255.0

If this is set you can now continue to start creating the drbd devices. If not check the output of /var/log/ha-log and try to figure out what is wrong.

 

4. DRBD Configuration

This manual implies that you have worked with drbd in the past and know howto tune the drbd common section regarding to your hardware. Using protocol C is save on stacked resources but not the best regarding network performance.

Remove the old drbd.conf and create a new one:

[email protected] ~# rm /etc/drbd.conf
[email protected] ~# nano /etc/drbd.conf
global {
        usage-count yes;
}
common {
        protocol C;
        disk {
                on-io-error detach;
                }
net {
        after-sb-0pri disconnect;
        after-sb-1pri disconnect;
        after-sb-2pri disconnect;
        rr-conflict disconnect;
        }
syncer {
  al-extents 257;
  rate 100M;
        }
handlers {
        pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
        pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
        local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
 }
 startup {
  degr-wfc-timeout 120; # 2 minutes.
 }
}
resource meta {
 on filer01 {
  device /dev/drbd0;
  disk /dev/sdb1;
  address 10.10.50.101:7788;
  meta-disk internal;
 }
 on filer02 {
  device /dev/drbd0;
  disk /dev/sdb1;
  address 10.10.50.102:7788;
  meta-disk internal;
 }
}
resource data {
 on filer01 {
  device /dev/drbd1;
  disk /dev/sdb2;
  address 10.10.50.101:7789;
  meta-disk internal;
 }
 on filer02 {
  device /dev/drbd1;
  disk /dev/sdb2;
  address 10.10.50.102:7789;
  meta-disk internal;
 }
}
resource meta-U {
 stacked-on-top-of meta {
  device /dev/drbd10;
  address 10.10.50.100:7788;
 }
 on filer03 {
  device /dev/drbd10;
  disk /dev/sdb1;
  address 10.10.50.103:7788;
  meta-disk internal;
 }
}
resource data-U {
 stacked-on-top-of data {
  device /dev/drbd11;
  address 10.10.50.100:7789;
 }
 on filer03 {
  device /dev/drbd11;
  disk /dev/sdb2;
  address 10.10.50.103:7789;
  meta-disk internal;
 }
}

Copy this file over to the other nodes:

[email protected] ~# scp /etc/drbd.conf [email protected]:/etc/drbd.conf
[email protected] ~# scp /etc/drbd.conf [email protected]:/etc/drbd.conf

Initiate the lower resources:

[email protected] ~# drbdadm create-md meta
[email protected] ~# drbdadm create-md data

[email protected] ~# drbdadm create-md meta
[email protected] ~# drbdadm create-md data

Start DRBD on filer01 and filer02:

[email protected] ~# service drbd start
[email protected] ~# service drbd start

Set the lower drbd resources primary on filer01:

[email protected] ~# drbdsetup /dev/drbd0 primary -o
[email protected] ~# drbdsetup /dev/drbd1 primary -o

Create the DRBD Metadata on the stacked ressource:

[email protected] ~# drbdadm --stacked create-md meta-U
[email protected] ~# drbdadm --stacked create-md data-U

Enable the stacked resource and set it to primary:

[email protected] ~# drbdadm --stacked up meta-U
[email protected] ~# drbdadm --stacked up data-U
[email protected] ~# drbdsetup /dev/drbd10 primary -o
[email protected] ~# drbdsetup /dev/drbd11 primary -o

Create DRBD metadata on filer03 and start the resources:

[email protected] ~# drbdadm create-md meta-U
[email protected] ~# drbdadm create-md data-U
[email protected] ~# service drbd start

Control the sync process on the nodes with:

[email protected] ~# service drbd status

drbd driver loaded OK; device status:
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by [email protected], 2010- 01-13 17:17:27
m:res cs ro ds p mounted fstyp e
0:meta Connected Primary/Secondary UpToDate/UpToDate C
1:data Connected Primary/Secondary UpToDate/UpToDate C
10:meta-U^^0 Connected Primary/Secondary UpToDate/UpToDate C
11:data-U^^1 Connected Primary/Secondary UpToDate/UpToDate C

This way it should look when all nodes are synced and filer01 is the primary.

 

5. Prepare The Configuration Partition

Filer01

Initiate meta partition:

NOTE: Here we gonna use the stacked resource instead of the lower level one!

[email protected] ~# mkfs.ext3 /dev/drbd10

Openfiler to Meta-Partition:

[email protected] ~# mkdir /meta
[email protected] ~# mount /dev/drbd10 /meta
[email protected] ~# mv /opt/openfiler/ /opt/openfiler.local
[email protected] ~# mkdir /meta/opt
[email protected] ~# cp -a /opt/openfiler.local /meta/opt/openfiler
[email protected] ~# ln -s /meta/opt/openfiler /opt/openfiler
[email protected] ~# rm /meta/opt/openfiler/sbin/openfiler
[email protected] ~# ln -s /usr/sbin/httpd /meta/opt/openfiler/sbin/openfiler
[email protected] ~# rm /meta/opt/openfiler/etc/rsync.xml
[email protected] ~# ln -s /opt/openfiler.local/etc/rsync.xml /meta/opt/openfiler/etc/
[email protected] ~# mkdir -p /meta/etc/httpd/conf.d

Samba/NFS/ISCSI/PROFTPD Configuration Files to Meta Partition:

[email protected] ~# service nfslock stop
[email protected] ~# service nfs stop
[email protected] ~# service rpcidmapd stop
[email protected] ~# umount -a -t rpc-pipefs
[email protected] ~# mkdir /meta/etc
[email protected] ~# mv /etc/samba/ /meta/etc/
[email protected] ~# ln -s /meta/etc/samba/ /etc/samba
[email protected] ~# mkdir -p /meta/var/spool
[email protected] ~# mv /var/spool/samba/ /meta/var/spool/
[email protected] ~# ln -s /meta/var/spool/samba/ /var/spool/samba
[email protected] ~# mkdir -p /meta/var/lib
[email protected] ~# mv /var/lib/nfs/ /meta/var/lib/
[email protected] ~# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[email protected] ~# mv /etc/exports /meta/etc/
[email protected] ~# ln -s /meta/etc/exports /etc/exports
[email protected] ~# mv /etc/ietd.conf /meta/etc/
[email protected] ~# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[email protected] ~# mv /etc/initiators.allow /meta/etc/
[email protected] ~# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[email protected] ~# mv /etc/initiators.deny /meta/etc/
[email protected] ~# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[email protected] ~# mv /etc/proftpd /meta/etc/
[email protected] ~# ln -s /meta/etc/proftpd/ /etc/proftpd

httpd Modules for Openfiler ( use lib64 for 64bit openfiler ):

[email protected] ~# rm /opt/openfiler/etc/httpd/modules
[email protected] ~# ln -s /usr/lib/httpd/modules /opt/openfiler/etc/httpd/modules

Restart Openfiler to see if everything worked:

[email protected] ~# service openfiler restart

NOTE: If it doesnt restart check if you have linked the httpd Modules, probably you use lib64 instead of lib!

 

Filer02 and Filer03

Openfiler Configuration:

mkdir /meta
mv /opt/openfiler/ /opt/openfiler.local
ln -s /meta/opt/openfiler /opt/openfiler

Samba/NFS/ISCSI/PROFTPD Configuration Files to Meta Partition:

service nfslock stop
service nfs stop
service rpcidmapd stop
umount -a -t rpc-pipefs
rm -rf /etc/samba/
ln -s /meta/etc/samba/ /etc/samba
rm -rf /var/spool/samba/
ln -s /meta/var/spool/samba/ /var/spool/samba
rm -rf /var/lib/nfs/
ln -s /meta/var/lib/nfs/ /var/lib/nfs
rm -rf /etc/exports
ln -s /meta/etc/exports /etc/exports
rm /etc/ietd.conf
ln -s /meta/etc/ietd.conf /etc/ietd.conf
rm /etc/initiators.allow
ln -s /meta/etc/initiators.allow /etc/initiators.allow
rm /etc/initiators.deny
ln -s /meta/etc/initiators.deny /etc/initiators.deny
rm -rf /etc/proftpd
ln -s /meta/etc/proftpd/ /etc/proftpd

Share this page:

3 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By:
Reply  

 Great guide but some mistakes on Page 1

services heartbeat start repair with
service heartbeat start

 Create etc/ha.cf/ ( filer01 and filer02 ):

 repair with

 Create etc/ha.d/ha.cf ( filer01 and filer02 ):

By: Wong
Reply  

hi, could anyone, i stack on here

[[email protected] ~]# drbdadm create-md meta
'meta' ignored, since this host (openfiler1.local) is not mentioned with an 'on' keyword.

thank you.

By: Mahesh
Reply  

Edit /etc/drbd.conf and use fully qualified hostname instead of short hostname. 

e.g.

resource data {
on san1.linux.com {