Installing and Configuring Openfiler with DRBD and Heartbeat

Introduction

Openfiler is a high performance operating system tailored for use as a SAN/NAS appliance. This configuration will enable two Openfiler appliances to work in an Active/Passive high availability scenario.

 

Requirements

Hardware

  • 2 x boxes that meet the minimum spec of Openfiler's hardware specifications.
  • 2 x ethernet interfaces in each box
  • Openfiler 2.3 installation media.
  • Both boxes should have the same size drives in each to avoid any replication inconsistencies.

 

Software

Install Openfiler 2.3 on both boxes utilizing a disk setup such as the following:

  • 3 GB root (“/”) partition
  • 2 GB “swap” partition
  • 512 MB “/meta” partition (used for DRBD0)
  • Data partition configured as an unmounted LVM (used for DRBD1)

 

Configuration

Network

Each Openfiler appliance will have two NICs: one for communicating with the LAN, the other for communicating with the
other SAN (via direct cable). The first will be used for administration, to communicate directly with each node.

A third “virtual” interface is used by the heartbeat service and is what will be used by computers on the LAN.

Below is what is used:

filer01

  • LAN Interface (eth0) 192.168.1.18
  • Replication Interface (eth1) 10.188.188.1

filer02

  • LAN Interface (eth0) 192.168.1.19
  • Replication Interface (eth1) 10.188.188.2

HA NAS/SAN Address (eth0) 192.168.1.17

  • This is configured in the cluster.xml file (do not attempt to configure anywhere else)

 

Hostname Setup

For both nodes to be able to recognize each other by name, configure the hosts file on each computer.
Modify our /etc/hosts (on filer01):

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer01 localhost.localdomain localhost
10.188.188.2 filer02

Modify our /etc/hosts (on filer02):

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 filer02 localhost.localdomain localhost
10.188.188.1 filer01

 

SSH Shared keys

To allow the two Openfiler appliances to talk to each other without having to use a password, use SSH shared keys.

On filer01:

root@filer01 ~# ssh-keygen -t dsa

Hit enter at the prompts (don't set a password on the key).

On filer02:

root@filer02 ~# ssh-keygen -t dsa

Hit enter at the prompts (don't set a password on the key).

The above command will generate a file called "id_dsa.pub" in ~/.ssh/, which is the public key that will need to be copied to
the other node:

root@filer01 ~# scp .ssh/id_dsa.pub root@filer02:~/.ssh/authorized_keys2
root@filer02 ~# scp .ssh/id_dsa.pub root@filer01:~/.ssh/authorized_keys2

 

Configure DRBD

DRBD is what will keep the data between the two nodes consistent.

On filer01:

root@filer01 ~# mv /etc/drbd.conf /etc/drbd.conf.org

Then modify drbd.conf (version 8) according to following:

global {
 # minor-count 64;
 # dialog-refresh 5; # 5 seconds
 # disable-ip-verification;
 usage-count ask;
}

common {
 syncer { rate 100M; }
}

resource cluster_metadata {
 protocol C;
 handlers {
  pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";
  pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";
  local-io-error "echo O > /proc/sysrq-trigger ; halt -f";
  # outdate-peer "/usr/sbin/drbd-peer-outdater";
 }
 
 startup {
  # wfc-timeout 0;
  degr-wfc-timeout 120; # 2 minutes.
 }

 disk {
  on-io-error detach;
 }

 net {
  after-sb-0pri disconnect;
  after-sb-1pri disconnect;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }

 syncer {
  # rate 10M;
  # after "r2";
  al-extents 257;
 }

 on filer01 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 10.188.188.1:7788;
  meta-disk internal;
 }

 on filer02 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 10.188.188.2:7788;
  meta-disk internal;
 }
}

resource vg0drbd {
 protocol C;
 startup {
  wfc-timeout 0; ## Infinite!
  degr-wfc-timeout 120; ## 2 minutes.
 }

 disk {
  on-io-error detach;
 }

 net {
  # timeout 60;
  # connect-int 10;
  # ping-int 10;
  # max-buffers 2048;
  # max-epoch-size 2048;
 }

 syncer {
  after "cluster_metadata";
 }

 on filer01 {
  device /dev/drbd1;
  disk /dev/sda5;
  address 10.188.188.1:7789;
  meta-disk internal;
 }

 on filer02 {
  device /dev/drbd1;
  disk /dev/sda5;
  address 10.188.188.2:7789;
  meta-disk internal;
 }
}

Both hosts need the same drbd.conf, so the drbd.conf file from filer01 will be copied to filer02:

root@filer01 ~# scp /etc/drbd.conf root@filer02:/etc/drbd.conf

Initialise metadata on /dev/drbd0 (cluster_metadata) and /dev/drbd1 (vg0drbd) on both nodes:

root@filer01 ~# drbdadm create-md cluster_metadata
root@filer01 ~# drbdadm create-md vg0drbd
root@filer02 ~# drbdadm create-md cluster_metadata
root@filer02 ~# drbdadm create-md vg0drbd

Note: if the commands above generate errors about needing to zero out the file system, use the following command:

root@filer01 ~# dd if=/dev/zero of=/dev/sda3

Be careful with this command and make sure its on the correct drive.

Before starting the DRBD service, make sure that the partition used for drbd0 (in the cluster_metadata resource in the drbd.conf file) is not already mounted (which it will be by default if it was created during the installation).

root@filer01 ~# umount /dev/sda3

Now, start DRBD on both hosts:

root@filer01 ~# service drbd start

root@filer02 ~# service drbd start

If all goes well, they should connect and running "service drbd status" should present output similar to the following:

root@filer1 /# service drbd status

drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by phil@mescal, 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
0:cluster_metadata Connected Secondary/Secondary Inconsistent/Inconsistent C
1:vg0drbd Connected Secondary/Secondary Inconsistent/Inconsistent C

Once both drbd resources are connected and both nodes are in Secondary state (as above), set a Primary node:

root@filer01 ~# drbdsetup /dev/drbd0 primary -o
root@filer01 ~# drbdsetup /dev/drbd1 primary -o

This should give you a status result of something like the following:

root@filer1 /# service drbd status

drbd driver loaded OK; device status:
version: 8.0.12 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by phil@mescal, 2008-04-24 13:29:44
m:res cs st ds p mounted fstype
... sync'ed: 17.9% (247232/297152)K
0:cluster_metadata SyncSource? Primary/Secondary UpToDate/Inconsistent C
1:vg0drbd PausedSyncS? Primary/Secondary UpToDate/Inconsistent C

Note: if the vg0drbd LVM is large, it will take a long time to sync (perhaps overnight).

Enable DRBD to startup at boot:

root@filer01 ~# chkconfig --level 2345 drbd on
root@filer02 ~# chkconfig --level 2345 drbd on

Now create the cluster_metadata filesystem. Use this 512 MB partition to keep all of the Openfiler configuration data and the data for the services that should be available in HA (eg. NFS, iSCSI, SMB).

root@filer01 ~# mkfs.ext3 /dev/drbd0

Don't add this partition to an /etc/fstab, as this is managed by Heartbeat (and will be configured shortly).

Share this page:

41 Comment(s)

Add comment

Comments

From: Anonymous at: 2008-12-24 03:25:30

I have some questions on HA. In this instruction, you have mention about IP address of HA (HA NAS/SAN Address (eth0) 192.168.1.17) so that mean we need to have another HA machine to do this or something else, please help to advise opn this.

Best regards,

Pisey

From: Ong Pisey at: 2008-12-25 11:08:11

I have completed all the instruction but except HA, I don't understand - where can we set the ip address of HA: 192.168.1.17, we set it on filer01, filer02 or ... something else. Please help on this.

Best regards,

Pisey

From: Adrian Sender at: 2009-01-08 12:09:31

The ip address you refer to is the "cluster" ip and sits on the primary node of the HA cluster.

In this example an ext3 filesystem is used; this means only one disk can be mounted at a particular instance; this node would have the cluster ip. It is possible to use gfs with drbd for a clustered filesystem & eliminate heartbeat for filesystem mounting and ip migration.

From: Ong Pisey at: 2009-01-09 09:50:34

I got it, now it is working. Thanks for your reply.

Best regards,

Pisey

From: Anonymous at: 2009-12-13 00:30:52

The GUI does not work after doing this. Especially setting up iScsi. Any ideas?

From: Ong Pisey at: 2009-01-09 10:10:27

Note: if the commands above generate errors about needing to zero out the file system, use the following command:

root@filer01 ~# dd if=/dev/zero of=/dev/sda3

it take long time to wait for this. For example if my partition is 1GB - i think it is ok but if it is 500GB. How long that I need to wait for run this command. So do we have another solution for this.

Thanks,

Pisey

From: Vitali at: 2009-02-18 08:05:10

You dont need to wait to complete operation.
you can also abort the command after some seconds.
in the fact you need only to delete the header of the partition.
i dont now how many bytes is mean.

From: Anonymous at: 2010-05-11 09:09:17

Or you can just wipe a few megabytes of the partition:

dd if=/dev/zero of=/dev/sda3 bs=1M count=128

From: Andres at: 2009-07-29 18:43:14

I really need to get this implementation done, but I have a little problem, I'm a real newbie on Linux, I have tested and used succesfully the OpenFiler Virtual Appliance, running it over ESX and it works fine, but I don't know how to modify the files you are saying. Is this configuration possible over the Virtual Appliance? or I need to do a clean install of Openfiler?


 

From: at: 2010-02-01 04:09:52

gilly05 and other folks,

This guide is great.

I was wondering if there is any other more recent and/or updated and/or revised version of this guide or anything with instructions how to set up DRBD and Heartbeat on OpenFiler 2.3.

Is there anything else?

 

Thanks,

 

PS: I have created a post for this in the forums:

http://www.howtoforge.com/forums/showthread.php?p=218075#post218075

From: Anonymous at: 2011-02-01 23:57:03

I found a tutorial that is for 2.3 the other day

http://wiki.the-mesh.org/wiki/OpenFilerHaSetup

 

 

From: Anonymous at: 2012-06-21 04:58:34

Dear all,

My drbd status show ro Primary/Secondary instead of st. What would be the issue ?

service drbd status

 

drbd driver loaded OK; device status:
version: 8.3.10 (api:86/proto:86)
GIT-hash: 5c9f89594553e32adb87d9638dce591782f947e3 build by phil@mescal, 2010-01-28 12:29:44
m:res cs ro ds p mounted fstype
0:cluster_metadata Connected Secondary/Secondary Inconsistent/Inconsistent C
1:vg0drbd Connected Secondary/Secondary Inconsistent/Inconsistent C

From: Anonymous at: 2009-04-13 19:57:06

Great article!

I would also like to add that if you are using iSCSI, the iscsi-target service must be restarted on the passive node every time you add a new LUN. Otherwise the passive node won't see the new LUN and won't failover correctly.

From: Anonymous at: 2009-03-27 17:12:55

udpport 694 must be before the bcast statement in the ha.cf. If you have more than one cluster setup, it will use the default 694 and you will get an error.

From: Anonymous at: 2008-10-26 18:46:35

There is a typo in:

/opt/openfiler.local/etc/rsync.xml

<remote hostname="10.188.1881"/> ## IP address of peer node.
should be:
<remote hostname="10.188.188.1"/> ## IP address of peer node.

Also:

<resource value="IPaddr::192.168.1.17/24" />
would be better expressed as:
<resource value="IPaddr::192.168.1.17/32" />
Good article.

From: PatrickD at: 2009-10-14 12:51:00

How do I set the HA IP address to a 3rd, physical NIC? Just add one and configure it with the desired IP?

From: Anonymous at: 2009-10-27 21:56:52

I have tried this like 20 times and it just doesn't work.  The /cluster_metadata never gets replicated to the second node and if the server is rebooted, /cluster_metadata won't mount.

 I followed the steps pricesly and the only error I gets is when trying to move the nfs directory (permissions issues) but that shouldn't cause the drives to not mount or stop replication.

 My guess is that the version 3.2 of OF is the issue.

 Thanks.

From: Rip at: 2009-01-12 11:35:46

Hi Gilly,

 Would prob be a good idea to throw up the credits to parts of the howto and some links on the openfiler forum that addresses some peoples issues.

 Sources: Eg. http://wiki.hyber.dk/doku.php and http://www.the-mesh.org

 

Cheers

Rip

From: vladdrac at: 2009-07-03 01:12:49

openfiler doesn't create haresources any idea ?

From: Anonymous at: 2009-07-21 00:25:26

Hi,

In order to make the haresources file, you need to login to Openfiler web interface and then you go to Services tab and you need to Enable some services. After that you will see haresources file in /etc/ha.d/

Pisey

From: kad at: 2010-06-29 05:16:28

attempt update pakage

conary updateall on both node

From: thefusa at: 2009-11-25 17:19:56

I get the same error....I've installed 20 times..same error..

 

From: Adam at: 2010-03-03 22:29:45

First make sure all NFS services are halted and umount rpc_pipefs:

service nfslock stop

service nfs stop

service rpcidmapd stop

umount -a -t rpc_pipefs


...Continued from his guide...

mkdir -p /cluster_metadata/var/lib

mv /var/lib/nfs/ /cluster_metadata/var/lib/

ln -s /cluster_metadata/var/lib/nfs/ /var/lib/nfs


Then one more thing:

mount -t rpc_pipefs sunrpc /var/lib/rpc_pipefs

service rpcidmapd start

From: at: 2010-03-11 13:42:04

Hi...

I configured my openfiler cluster with exactly like this article.

But when i reboot the both filer, i get this error on filer01:

Checking filesystems

/1: clean, 30108/767232 files, 170424/767095 blocks

fsck.ext3: Unable to resolve 'LABEL=/meta'

And I get this error on filer02:

/1: clean, 30108/767232 files, 170424/767095 blocks

fsck.ext3: Unable to resolve 'LABEL=/meta1'

Can anyone help me??? I'm just dispairing. THX

From: sgi at: 2010-03-19 03:54:46

Maya,

 from the recovery console try:

 e2label /dev/xxx /meta

 (where xxx is your device mounted /meta)

 ~sgi

From: maya at: 2010-04-27 13:19:09

Hi sgi

You are the best!

 Thank you very much. This resovled my problem!

 Maya

From: Didier at: 2010-06-11 01:18:28

I'm having the same problem, and even though I add /dev/xxx /data still the problem persist.

This is my /etc/fstab file:

LABEL=/                 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext2    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
LABEL=/meta1            /meta                   ext3    defaults        1 2
/proc                   /proc                   proc    defaults        0 0
/sys                    /sys                    sysfs   defaults        0 0
LABEL=SWAP-hda3         swap                    swap    defaults        0 0

At the recovery console I type:

e2label /dev/hda5 /meta1

After that I reboot and the problem persist, then I try with the following

e2label /dev/hda5 /meta

After that I reboot and the problem persist.

If I comment the line in the fstab file the system boots fine but Openfiler does not start, complaining that it cannot find /opt/openfiler/etc/httpd/conf/httpd.conf: No such file or directory
/b in/bash: /opt/openfiler/sbin/openfiler: No such file or directory

This is a sample of my /etc/drbd.conf file
resource meta {
        device          /dev/drbd0;
        disk            /dev/hda5;

I have follow the procedure many times already, everything works fine until I reboot the systems.

Any help is appreciated

From: Anonymous at: 2010-10-08 22:01:13

I had the same issue.  I replaced LABEL=/meta with the actual disk information. Ex; /dev/sdb5.  I also added noauto to the options.  My line reads:

 /dev/sdb5               /meta                   ext3    defaults,noauto 1 2

Things worked fine after that.

From: Anonymous at: 2010-10-27 05:06:34

I did the same but once I reboot filer 1 and 2 the cluster_metadata /dev/sda5 is not mounted and I could not login to the HA IP address

 

Please help!

From: Anonymous at: 2011-01-18 13:13:25

I got the same problems, tryed to changed the fstab 'LABEL=/meta' entry to point to the disk partition and all other tips in the comments and they did not work.


From: Ben at: 2011-08-11 23:40:30

Just redo the boxes and after u partition the drives within the install, nano /etc/fstab and remove the /meta line form the file

From: santosh at: 2011-09-08 15:22:29

A tutorial about how-to Configure Openfiler 2.99 as a iSCSI target for VMware vSphere 5 (ESXi 5). This will help all users/new newbie,

http://www.mytricks.in/2011/09/guide-installing-openfiler-on-vmware.html
http://www.mytricks.in/2011/09/guide-configure-openfiler-299-with.html
http://www.mytricks.in/2011/09/guide-configure-openfiler-299-with_08.html

From: at: 2010-04-09 03:49:54

Some of the resource lines are missing the tag ending />

ie

<resource value="drbddisk::">
<resource value="LVM::vg0drbd">
<resource value="Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime">

should be

<resource value="drbddisk::" />
<resource value="LVM::vg0drbd" />
<resource value="Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime" />


The rsync code first parses cluster.xml to work out if clusting is enabled but falls over because the tags are not consistent (no errors are displayed on the openfiler config pages.)
Since it can't determine if clustering is enabled the rsync process never happens. This means all files listed in rsync.xml file never sync between nodes until the tags are fixed in cluster.xml

From: at: 2010-06-02 06:08:10

the entry

udpport 694

should go before the entry

bcast eth1

This above config will work fine as it will always default to port 694 if the udpport entry is in the wrong order. For those who have two HA clusters (like me) you need to have a different port number for each cluster which means you need the udpport line before the bcast line.

From: Rombik at: 2010-06-24 14:15:04

Help pls.,

After several minutes it is unmounted /cluster_metadata 

After restart of service  heartbeat on any node -
 is again mounted and again unmounted 


 cluster node also ping\no ping

(heartbeat and drbd service - started )

 for ex. 

[root@filer01 ~]# service heartbeat status
heartbeat OK [pid 16249 et al] is running on filer01 [filer01]...

and node 2 - started too.

From: Anonymous at: 2010-10-08 21:58:37

Hello,

 I've followed the instructions, and everything seems to be working well.  The problem I have is with configuring volumes after DRBD & heartbeat are configured.

 For my initial disk configurations I used /dev/sdb (local storage) and those failover properly.  However, I would now like to configure /dev/sda (Fiber RAID5 storage).  I set everything up in Openfiler, but when I perform a test failover (/usr/lib/heartbeat/hb_standby on the primary node) I don't see the volume I just created on the primary node.

Can someone explain the process a little bit more clearly with regards to setting up volumes in this environment?  Am I missing something?

Thanks,

  -Josh

From: at: 2011-01-13 10:40:59

Hi Techies,

Nice Post I was also looking for mysql replication clustering , oracle clustering along with also implementation of DRBD,High Availability using Heartbeat  too and found a great ebook on http://www.ebooksyours.com/what-is-clustering.html and this ebook was a complete worthy purchasing as it consisted of complete thorough implementation of all clustering technologies with live examples and configurations.

 Cheers !

Akki

From: at: 2011-02-06 04:27:35

In my previous post, I supplied the wrong cluster.xml file. Here is the correct cluster.xml file. Heartbeat will create two HA IP addresses, one on each NIC (i.e. eth0 and eth1).

 <?xml version="1.0" ?>
<cluster>
<clustering state="on" />
<nodename value="san01" />
<resource value="MailTo::itsupport@co2crc.com.au::ClusterFailover"/>
<resource value="IPaddr::192.168.0.6/24/eth0:0" />
<resource value="IPaddr::192.168.7.16/24/eth1:0" />
<resource value="drbddisk::">
<resource value="LVM::vg0drbd">
<resource value="Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime">
<resource value="MakeMounts"/>
</cluster>

From: at: 2011-02-05 12:49:01

Hi, this is if for anyone who finds the below comments of interest.  First, thanks to all who created Openfiler, those who wrote this HOWTO, and also those who contributed ideas or questions here, and in the Openfiler forums.

After I had created a VMware ESX HA demonstration setup in a i7 CPU, whitebox ESXi server which used a single Openfiler iSCSI SAN, I wanted to test out the above HOWTO on setting up an Openfiler Cluster. My interest was in an iSCSI SAN Cluster. I built this cluster as two Openfiler guests in the i7 ESXi host server.  Because of this I am unable to accurately test performance. Soon I want to reproduce this with real hardware, not virtual.  The real question I am trying to solve is how to have an effective Openfiler SAN HA Cluster for VMware using only 1 GB NICs (sadly 10 GB NICs are currently too expensive for my budget).

Reading VMware documentation, it is my understanding that VMware does not use NIC  teaming to make 2, 1GB NICs into a single 2 GB bond (please tell me how, if I am wrong).  It is also my understanding that VMware's best and recommend approach is to create two VMKernels with one NIC each, then on your SAN with two NICs create two paths/networks to the SAN's iSCSI targets. When you have two (or more) paths in vSphere, you can then select "Round Robin (VMware)"for the iSCSI targets.  I have tested this and it worked well, giving good reliable data transfer, but not faster network speed, why?, well because each NIC/Path is still only 1 Gbps each, and are used in turn not together at the same time. When also used with two separate Gb switches one for each path/network, it does allow for increased fault tolerance by having a redundant path.

This configuration was possible once I learned that I could specify two HA IP addresses using "IPaddr::192.168.0.6/24/eth0:0 IPaddr::192.168.7.16/24/eth1:0 ".  Using the Cluster built from this HOWTO, I used "ifconfig" to display the IP settings which showed that the IP address for the HA IP Address was configured as eth0:0 which gave me the idea of using "192.168.0.6/24/eth0:0" in the Cluster.xml file and then I created a second IP address entry for the eth1 as "IPaddr::192.168.7.16/24/eth1:0".

The simplest way I could think of explaining how to configured this, is to show you a few of the important configuration files.

Two Openfiler servers called san01 and san02 (just to use a different name than filer0x). Each server has 4 NICs and one hard drive.
The hard disk drive partitioning that I used is slightly different to the above HOWTO:

  • 3072 MB root (“/”) partition
  • 3072 MB log (“/var/log”) partition
  • 2048 MB “swap” partition
  • 1024 MB “/meta” partition (used for DRBD0)
  • The remainder of the drive as an unmounted LVM (used for DRBD1)

root@san01 ~]# fdisk -l /dev/sda

Disk /dev/sda: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 392 3148708+ 83 Linux
/dev/sda2 393 784 3148740 83 Linux
/dev/sda3 785 1045 2096482+ 82 Linux swap / Solaris
/dev/sda4 1046 13054 96462292+ 5 Extended
/dev/sda5 1046 1176 1052226 83 Linux
/dev/sda6 1177 13054 95410003+ 8e Linux LVM
[root@san01 ~]#

The four NICs are configured as follows;
(bond0 is eth2 and eth3 bonded together)
High Availability IP address for eth0 is 192.168.0.6
High Availability IP address for eth1 is 192.168.7.16

 (text below is copied from ifconfig output);

san01
bond0     inet addr:192.168.5.1  Bcast:192.168.5.255  Mask:255.255.255.0
eth0      inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
eth0:0    inet addr:192.168.0.6  Bcast:192.168.0.255  Mask:255.255.255.0
eth1      inet addr:192.168.7.11  Bcast:192.168.7.255  Mask:255.255.255.0
eth1:0    inet addr:192.168.7.16  Bcast:192.168.7.255  Mask:255.255.255.0
eth2      inet addr:192.168.5.1  Bcast:192.168.5.255  Mask:255.255.255.0
eth3      inet addr:192.168.5.1  Bcast:192.168.5.255  Mask:255.255.255.0

san02
bond0     inet addr:192.168.5.2  Bcast:192.168.5.255  Mask:255.255.255.0
eth0      inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
eth0:0    inet addr:192.168.0.6  Bcast:192.168.0.255  Mask:255.255.255.0
eth1      inet addr:192.168.7.12  Bcast:192.168.7.255  Mask:255.255.255.0
eth1:0    inet addr:192.168.7.16  Bcast:192.168.7.255  Mask:255.255.255.0
eth2      inet addr:192.168.5.2  Bcast:192.168.5.255  Mask:255.255.255.0
eth3      inet addr:192.168.5.2  Bcast:192.168.5.255  Mask:255.255.255.0

The /cluster_metadata/opt/openfiler/etc/cluster.xml file is;
<?xml version="1.0" ?>
<cluster>
<clustering state="on" />
<nodename value="filer01" />
<resource value="MailTo::itsupport@co2crc.com.au::ClusterFailover"/>
<resource value="IPaddr::192.168.0.6/24/eth0" />
<resource value="IPaddr::192.168.0.16/24/eth4" />
<resource value="drbddisk::">
<resource value="LVM::vg0drbd">
<resource value="Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime">
<resource value="MakeMounts"/>
</cluster>

Which created /etc/ha.d/haresources
san01 MailTo::itsupport@co2crc.com.au::ClusterFailover IPaddr::192.168.0.6/24/eth0:0 IPaddr::192.168.7.16/24/eth1:0 drbddisk:: LVM::vg0drbd Filesystem::/dev/drbd0::/cluster_metadata::ext3::defaults,noatime MakeMounts iscsi-target openfiler

This is the drbd status

[root@san01 ~]# service drbd status ; date
drbd driver loaded OK; device status:
version: 8.2.7 (api:88/proto:86-88)
GIT-hash: 61b7f4c2fc34fe3d2acf7be6bcc1fc2684708a7d build by phil@fat-tyre, 2008-11-12 16:47:11
m:res cs st ds p mounted fstype
0:cluster_metadata Connected Primary/Secondary UpToDate/UpToDate C /cluster_metadata ext3
1:vg0drbd Connected Primary/Secondary UpToDate/UpToDate C
Fri Feb 4 19:40:25 EST 2011
[root@san01 ~]#

 

I am curious if anyone has any further ideas or useful comments.

From: Anonymous at: 2011-09-20 15:29:29

ESX will do nic teaming to make a single trunk.

1) get a smart switch (capable of link aggregation) and an ESX server with multiple nics.
2) in vSphere, go to the server configuration of the ESX box -> networking -> find your VMNetwork and click Properties.
3) Add the unassigned NIC's to the VM Network vswitch.
4) Select properties on the VSwitch -> setup your link aggregation there
5) (AFTER YOU HAVE DONE 4, NOT BEFORE) turn link aggregation on in your switch
6) ???
7) profit

From: Jon at: 2011-11-17 22:40:21

If you're setting this up with more than one data partition/volume group, the LV created at the end of the tutorial must be made on each volume group or heartbeat will fail.  

 ex:

 root@filer01 ~# lvcreate -L 400M -n StartVol_0 vg0drbd

 root@filer01 ~# lvcreate -L 400M -n StartVol_1 vg1drbd

 root@filer01 ~# lvcreate -L 400M -n StartVol_2 vg2drbd

 root@filer01 ~# lvcreate -L 400M -n StartVol_3 vg3drbd