On this page
- Openfiler 2.99 Active/Passive With Corosync, Pacemaker And DRBD
- 1. Create Systems with following setup:
- 2. Create meta/data Partition on both filers
- 2.1 Create DRBD Setup
- 2.2 Prepare the Configuration Partition
- 2.2.1 Openfiler to meta-Partition
- 2.2.2 Samba/NFS/ISCSI/PROFTPD Configuration Files to Meta Partition
- 2.2.3 httpd Modules for Openfiler
- 2.2.4 filer02 Openfiler Configuration
- 2.2.5 Samba/NFS/ISCSI/ProFTPD Configuration Files to Meta Partition
- 2.3 Prepare the Data Partition
- 3. Start Corosync and create a configuration for it:
Openfiler 2.99 Active/Passive With Corosync, Pacemaker And DRBD
Openfiler is a Linux based NAS/SAN Application which can deliver storage over nfs/smb/iscsi and ftp. It has a web interface over that you can control these services. This howto is based on the latest version of openfiler at this date, you can download it from the official homepage www.openfiler.com.
Thanks to the Openfiler team that made this howto possible.
1. Create Systems with following setup:
- hostname: filer01
- eth0: 10.10.11.101
- eth1: 10.10.50.101
- 500MB Meta partition
- 4GB+ Data partition
- hostname: filer02
- eth0: 10.10.11.102
- eth1: 10.10.50.102
- 500MB Meta partition
- 4GB+ Data partition
virtualip: 10.10.11.105 ( don't use on any adapter, we will make this later with corosync )
1.1 Create hosts file for easier access
[email protected] ~# nano /etc/hosts
Add:
10.10.50.102 filer02
[email protected] ~# nano /etc/hosts
On filer02 add:
10.10.50.101 filer01
1.2 Create/Exchange SSH Keys for easier file exchange
[email protected] ~# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
Do the same on filer02.
[email protected] ~# ssh-keygen -t dsa
Then exchange the files:
[email protected] ~# scp ~/.ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys
[email protected] ~# scp ~/.ssh/id_dsa.pub [email protected]:~/.ssh/authorized_keys
And now you can exchange files between the nodes without entering a password.
2. Create meta/data Partition on both filers
Before we can actually start the cluster we have to prepaire both systems and let the data and meta partition sync before it can be used by corosync/pacemaker as the first cluster config will start drbd and take over the control of this service. So we prepaire our partitions this time before we do the actual cluster configuration as we did in openfiler 2.3.
2.1 Create DRBD Setup
Edit /etc/drbd.conf on filer01 and filer02:
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; #include "drbd.d/*.res"; resource meta { on filer01 { device /dev/drbd0; disk /dev/sdb1; address 10.10.50.101:7788; meta-disk internal; } on filer02 { device /dev/drbd0; disk /dev/sdb1; address 10.10.50.102:7788; meta-disk internal; } } resource data { on filer01 { device /dev/drbd1; disk /dev/sdb2; address 10.10.50.101:7789; meta-disk internal; } on filer02 { device /dev/drbd1; disk /dev/sdb2; address 10.10.50.102:7789; meta-disk internal; } }
After that create the meta-data on it, if you get errors when this happens, please empty out the filesystem with, if you have anything in /etc/fstab related to the partitions /meta then remove these lines. ( This happens when you create the meta partitions in the installation phase ).
dd if=/dev/zero of=/dev/drbdX
[email protected] ~# drbdadm create-md meta
[email protected] ~# drbdadm create-md data
[email protected] ~# drbdadm create-md meta
[email protected] ~# drbdadm create-md data
Now you can start up drbd with:
service drbd start
on both nodes.
Make one node primary:
[email protected] ~# drbdsetup /dev/drbd0 primary -o
[email protected] ~# drbdsetup /dev/drbd1 primary -o
2.2 Prepare the Configuration Partition
[email protected] ~# mkfs.ext3 /dev/drbd0
[email protected] ~# service openfiler stop
2.2.1 Openfiler to meta-Partition
[email protected] ~# mkdir /meta
[email protected] ~# mount /dev/drbd0 /meta
[email protected] ~# mv /opt/openfiler/ /opt/openfiler.local
[email protected] ~# mkdir /meta/opt
[email protected] ~# cp -a /opt/openfiler.local /meta/opt/openfiler
[email protected] ~# ln -s /meta/opt/openfiler /opt/openfiler
[email protected] ~# rm /meta/opt/openfiler/sbin/openfiler
[email protected] ~# ln -s /usr/sbin/httpd /meta/opt/openfiler/sbin/openfiler
[email protected] ~# rm /meta/opt/openfiler/etc/rsync.xml
[email protected] ~# ln -s /opt/openfiler.local/etc/rsync.xml /meta/opt/openfiler/etc/
[email protected] ~# mkdir -p /meta/etc/httpd/conf.d
2.2.2 Samba/NFS/ISCSI/PROFTPD Configuration Files to Meta Partition
[email protected] ~# service nfslock stop
[email protected] ~# umount -a -t rpc-pipefs
[email protected] ~# mkdir /meta/etc
[email protected] ~# mv /etc/samba/ /meta/etc/
[email protected] ~# ln -s /meta/etc/samba/ /etc/samba
[email protected] ~# mkdir -p /meta/var/spool
[email protected] ~# mv /var/spool/samba/ /meta/var/spool/
[email protected] ~# ln -s /meta/var/spool/samba/ /var/spool/samba
[email protected] ~# mkdir -p /meta/var/lib
[email protected] ~# mv /var/lib/nfs/ /meta/var/lib/
[email protected] ~# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[email protected] ~# mv /etc/exports /meta/etc/
[email protected] ~# ln -s /meta/etc/exports /etc/exports
[email protected] ~# mv /etc/ietd.conf /meta/etc/
[email protected] ~# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[email protected] ~# mv /etc/initiators.allow /meta/etc/
[email protected] ~# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[email protected] ~# mv /etc/initiators.deny /meta/etc/
[email protected] ~# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[email protected] ~# mv /etc/proftpd /meta/etc/
[email protected] ~# ln -s /meta/etc/proftpd/ /etc/proftpd
2.2.3 httpd Modules for Openfiler
[email protected] ~# rm /opt/openfiler/etc/httpd/modules
[email protected] ~# ln -s /usr/lib64/httpd/modules /opt/openfiler/etc/httpd/modules
Now do a start and see if Openfiler can run:
[email protected] ~# service openfiler start
2.2.4 filer02 Openfiler Configuration
[email protected] ~# service openfiler stop
[email protected] ~# mkdir /meta
[email protected] ~# mv /opt/openfiler/ /opt/openfiler.local
[email protected] ~# ln -s /meta/opt/openfiler /opt/openfiler
2.2.5 Samba/NFS/ISCSI/ProFTPD Configuration Files to Meta Partition
[email protected] ~# service nfslock stop
[email protected] ~# umount -a -t rpc-pipefs
[email protected] ~# rm -rf /etc/samba/
[email protected] ~# ln -s /meta/etc/samba/ /etc/samba
[email protected] ~# rm -rf /var/spool/samba/
[email protected] ~# ln -s /meta/var/spool/samba/ /var/spool/samba
[email protected] ~# rm -rf /var/lib/nfs/
[email protected] ~# ln -s /meta/var/lib/nfs/ /var/lib/nfs
[email protected] ~# rm -rf /etc/exports
[email protected] ~# ln -s /meta/etc/exports /etc/exports
[email protected] ~# rm /etc/ietd.conf
[email protected] ~# ln -s /meta/etc/ietd.conf /etc/ietd.conf
[email protected] ~# rm /etc/initiators.allow
[email protected] ~# ln -s /meta/etc/initiators.allow /etc/initiators.allow
[email protected] ~# rm /etc/initiators.deny
[email protected] ~# ln -s /meta/etc/initiators.deny /etc/initiators.deny
[email protected] ~# rm -rf /etc/proftpd
[email protected] ~# ln -s /meta/etc/proftpd/ /etc/proftpd
2.3 Prepare the Data Partition
Change the lvm filter in the
/etc/lvm/lvm.conf
file from:
filter = [ "a/.*/" ]
to
filter = [ "a|drbd[0-9]|", "r|.*|" ]
Exchange this file to the other filer node
[email protected] ~# scp /etc/lvm/lvm.conf [email protected]:/etc/lvm/lvm.conf
After that we can create the actual used stuff:
[email protected] ~# pvcreate /dev/drbd1
[email protected] ~# vgcreate data /dev/drbd1
[email protected] ~# lvcreate -L 400M -n filer data
3. Start Corosync and create a configuration for it:
3.1 Create Corosync authkey
[email protected]~# corosync-keygen
( Press the real keyboard instead of pressing keys in an ssh terminal. )
Copy the authkey file to the other node and change the fileaccess:
[email protected]~# scp /etc/corosync/authkey [email protected]:/etc/corosync/authkey
[email protected]~# chmod 400 /etc/corosync/authkey
3.2 Create a file named pcmk /etc/corosync/service.d/pcmk
[email protected]~# vi /etc/corosync/service.d/pcmk
service { # Load the Pacemaker Cluster Resource Manager name: pacemaker ver: 0 }
3.2.1 Exchange this file to the other node
[email protected]~# scp /etc/corosync/service.d/pcmk [email protected]:/etc/corosync/service.d/pcmk
3.3 Create the corosync.conf file and change it to present your lan net ( bindnetaddr )
[email protected]~# vi /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page compatibility: whitetank totem { version: 2 secauth: off threads: 0 interface { ringnumber: 0 bindnetaddr: 10.10.50.0 mcastaddr: 226.94.1.1 mcastport: 5405 ttl: 1 } } logging { fileline: off to_stderr: no to_logfile: yes to_syslog: yes logfile: /var/log/cluster/corosync.log debug: off timestamp: on logger_subsys { subsys: AMF debug: off } } amf { mode: disabled }
3.3.1 Exchange the file to the other node
[email protected]~# scp /etc/corosync/corosync.conf [email protected]:/etc/corosync/corosync.conf