Installing and Configuring Openfiler with DRBD and Heartbeat

Introduction

Openfiler is a high performance operating system tailored for use as a SAN/NAS appliance. This configuration will enable two Openfiler appliances to work in an Active/Passive high availability scenario.

 

Requirements

Hardware

  • 2 x boxes that meet the minimum spec of Openfiler's hardware specifications.
  • 2 x ethernet interfaces in each box
  • Openfiler 2.3 installation media.
  • Both boxes should have the same size drives in each to avoid any replication inconsistencies.

 

Software

Install Openfiler 2.3 on both boxes utilizing a disk setup such as the following:

  • 3 GB root (“/”) partition
  • 2 GB “swap” partition
  • 512 MB “/meta” partition (used for DRBD0)
  • Data partition configured as an unmounted LVM (used for DRBD1)

 

Configuration

Network

Each Openfiler appliance will have two NICs: one for communicating with the LAN, the other for communicating with the
other SAN (via direct cable). The first will be used for administration, to communicate directly with each node.

A third “virtual” interface is used by the heartbeat service and is what will be used by computers on the LAN.

Below is what is used:

filer01

  • LAN Interface (eth0) 192.168.1.18
  • Replication Interface (eth1) 10.188.188.1

filer02

  • LAN Interface (eth0) 192.168.1.19
  • Replication Interface (eth1) 10.188.188.2

HA NAS/SAN Address (eth0) 192.168.1.17

  • This is configured in the cluster.xml file (do not attempt to configure anywhere else)

 

Hostname Setup

For both nodes to be able to recognize each other by name, configure the hosts file on each computer.
Modify our /etc/hosts (on filer01):

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer01 localhost.localdomain localhost
10.188.188.2 filer02

Modify our /etc/hosts (on filer02):

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer02 localhost.localdomain localhost
10.188.188.1 filer01

 

SSH Shared keys

To allow the two Openfiler appliances to talk to each other without having to use a password, use SSH shared keys.

On filer01:

  [email protected] ~# ssh-keygen -t dsa

Hit enter at the prompts (don't set a password on the key).

On filer02:

  [email protected] ~# ssh-keygen -t dsa

Hit enter at the prompts (don't set a password on the key).

The above command will generate a file called "id_dsa.pub" in ~/.ssh/, which is the public key that will need to be copied to
the other node:

[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys2
[email protected] ~# scp .ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys2

 

Configure DRBD

DRBD is what will keep the data between the two nodes consistent.

On filer01:

  [email protected] ~# mv /etc/drbd.conf /etc/drbd.conf.org

Then modify drbd.conf (version 8) according to following:

global {
 # minor-count 64;
 # dialog-refresh 5; # 5 seconds
 # disable-ip-verification;
 usage-count ask;
}

common {
 syncer { rate 100M; }
}

resource cluster_metadata {
 protocol C;
 handlers {
  pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
  pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
  local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
  # outdate-peer "/usr/sbin/drbd-peer-outdater";
 }
 
 startup {
  # wfc-timeout 0;
  degr-wfc-timeout 120; # 2 minutes.
 }

 disk {
  on-io-error detach;
 }

 net {
  after-sb-0pri disconnect;
  after-sb-1pri disconnect;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }

 syncer {
  # rate 10M;
  # after "r2";
  al-extents 257;
 }

 on filer01 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 10.188.188.1:7788;
  meta-disk internal;
 }

 on filer02 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 10.188.188.2:7788;
  meta-disk internal;
 }
}

resource vg0drbd {
 protocol C;
 startup {
  wfc-timeout 0; ## Infinite!
  degr-wfc-timeout 120; ## 2 minutes.
 }

 disk {
  on-io-error detach;
 }

 net {
  # timeout 60;
  # connect-int 10;
  # ping-int 10;
  # max-buffers 2048;
  # max-epoch-size 2048;
 }

 syncer {
  after "cluster_metadata";
 }

 on filer01 {
  device /dev/drbd1;
  disk /dev/sda5;
  address 10.188.188.1:7789;
  meta-disk internal;
 }

 on filer02 {
  device /dev/drbd1;
  disk /dev/sda5;
  address 10.188.188.2:7789;
  meta-disk internal;
 }
}

Both hosts need the same drbd.conf, so the drbd.conf file from filer01 will be copied to filer02:

  [email protected] ~# scp /etc/drbd.conf [email protected]:/etc/drbd.conf

Initialise metadata on /dev/drbd0 (cluster_metadata) and /dev/drbd1 (vg0drbd) on both nodes:

[email protected] ~# drbdadm create-md cluster_metadata
[email protected] ~# drbdadm create-md vg0drbd
[email protected] ~# drbdadm create-md cluster_metadata
[email protected] ~# drbdadm create-md vg0drbd

Note: if the commands above generate errors about needing to zero out the file system, use the following command:

  [email protected] ~# dd if=/dev/zero of=/dev/sda3

Be careful with this command and make sure its on the correct drive.

Before starting the DRBD service, make sure that the partition used for drbd0 (in the cluster_metadata resource in the drbd.conf file) is not already mounted (which it will be by default if it was created during the installation).

  [email protected] ~# umount /dev/sda3

Now, start DRBD on both hosts:

[email protected] ~# service drbd start
  [email protected] ~# service drbd start

If all goes well, they should connect and running "service drbd status" should present output similar to the following:

  [email protected] /# service drbd status

drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by [email protected], 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
0:cluster_metadata Connected Secondary/Secondary Inconsistent/Inconsistent C
1:vg0drbd Connected Secondary/Secondary Inconsistent/Inconsistent C

Once both drbd resources are connected and both nodes are in Secondary state (as above), set a Primary node:

[email protected] ~# drbdsetup /dev/drbd0 primary -o
[email protected] ~# drbdsetup /dev/drbd1 primary -o

This should give you a status result of something like the following:

  [email protected] /# service drbd status

drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by [email protected], 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
... sync'ed: 17.9% (247232/297152)K
0:cluster_metadata SyncSource? Primary/Secondary UpToDate/Inconsistent C
1:vg0drbd PausedSyncS? Primary/Secondary UpToDate/Inconsistent C

Note: if the vg0drbd LVM is large, it will take a long time to sync (perhaps overnight).

Enable DRBD to startup at boot:

[email protected] ~# chkconfig --level 2345 drbd on
[email protected] ~# chkconfig --level 2345 drbd on

Now create the cluster_metadata filesystem. Use this 512 MB partition to keep all of the Openfiler configuration data and the data for the services that should be available in HA (eg. NFS, iSCSI, SMB).

  [email protected] ~# mkfs.ext3 /dev/drbd0

Don't add this partition to an /etc/fstab, as this is managed by Heartbeat (and will be configured shortly).

Share this page:

12 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By: Anonymous

I have some questions on HA. In this instruction, you have mention about IP address of HA (HA NAS/SAN Address (eth0) 192.168.1.17) so that mean we need to have another HA machine to do this or something else, please help to advise opn this.

Best regards,

Pisey

By: Ong Pisey

I have completed all the instruction but except HA, I don't understand - where can we set the ip address of HA: 192.168.1.17, we set it on filer01, filer02 or ... something else. Please help on this.

Best regards,

Pisey

By: Adrian Sender

The ip address you refer to is the "cluster" ip and sits on the primary node of the HA cluster.

In this example an ext3 filesystem is used; this means only one disk can be mounted at a particular instance; this node would have the cluster ip. It is possible to use gfs with drbd for a clustered filesystem & eliminate heartbeat for filesystem mounting and ip migration.

By: Ong Pisey

I got it, now it is working. Thanks for your reply.

Best regards,

Pisey

By: Anonymous

The GUI does not work after doing this. Especially setting up iScsi. Any ideas?

By: Ong Pisey

Note: if the commands above generate errors about needing to zero out the file system, use the following command:

[email protected]filer01 ~# dd if=/dev/zero of=/dev/sda3

it take long time to wait for this. For example if my partition is 1GB - i think it is ok but if it is 500GB. How long that I need to wait for run this command. So do we have another solution for this.

Thanks,

Pisey

By: Vitali

You dont need to wait to complete operation.
you can also abort the command after some seconds.
in the fact you need only to delete the header of the partition.
i dont now how many bytes is mean.

By: Anonymous

Or you can just wipe a few megabytes of the partition:

dd if=/dev/zero of=/dev/sda3 bs=1M count=128

By: Andres

I really need to get this implementation done, but I have a little problem, I'm a real newbie on Linux, I have tested and used succesfully the OpenFiler Virtual Appliance, running it over ESX and it works fine, but I don't know how to modify the files you are saying. Is this configuration possible over the Virtual Appliance? or I need to do a clean install of Openfiler?


 

By:

gilly05 and other folks,

This guide is great.

I was wondering if there is any other more recent and/or updated and/or revised version of this guide or anything with instructions how to set up DRBD and Heartbeat on OpenFiler 2.3.

Is there anything else?

 

Thanks,

 

PS: I have created a post for this in the forums:

https://www.howtoforge.com/forums/showthread.php?p=218075#post218075

By: Anonymous

I found a tutorial that is for 2.3 the other day

http://wiki.the-mesh.org/wiki/OpenFilerHaSetup

 

 

By: Anonymous

Dear all,

My drbd status show ro Primary/Secondary instead of st. What would be the issue ?

service drbd status

 

drbd driver loaded OK; device status:
version: 8.3.10 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by [email protected], 2010-01-28 12:29:44
m:res cs ro ds p mounted fstype
0:cluster_metadata Connected Secondary/Secondary Inconsistent/Inconsistent C
1:vg0drbd Connected Secondary/Secondary Inconsistent/Inconsistent C