Setting Up An Active/Active Samba CTDB Cluster Using GFS & DRBD (CentOS 5.5)
This article explains how to set up an Active/Active Samba CTDB Cluster, using GFS and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.
Introduction
We use two nodes, both are active nodes, and the requests are loadbalanced. To replicate data between the nodes we use DRBD. To active/active cluster, we must use a ClusterFS (in this case GFS), to make the two nodes to write on the DRBD resource at same time.
1 Preliminary Note
Linux Distribution:
We are using the CentOS 5.5 (64bits) distribution, but will probably work on Fedora (and Red Hat, for sure). The installation of the CentOS is very simple and classical, select the base packages and other stuff that you like/need. One issue that must be remembered is that we use DRBD to replicate data between the nodes, then you'll need to have a disk or partition exclusive to DRBD. Remember this before partitioning disks on CentOS installation.
Network Hardware/Topology:
We use two Gigabit NIC's per node, one (eth0) connect to the network (LAN), and the other one (eth1) with a cross-over cable connecting both nodes.
The cross-over cable must be used to improve performance and confiability of the system, because DRBD won't depends of network switchs or anything else to replicate data between the nodes.
In this tutorial we will use the physical nodes node1.clusterbr.int and node2.clusterbr.int:
node1.clustersmb.int: Uses IP 10.0.0.181 (LAN) and IP 172.16.0.1 (cross-over)
node2.clustersmb.int: Uses IP 10.0.0.182 (LAN) and IP 172.16.0.2 (cross-over)
vip1.clustersmb.int: It's the Cluster IP, from node1 10.0.0.183.
vip2.clustersmb.int: It's the Cluster IP, from node2 10.0.0.184.
Disks:
Both the nodes have two disks:
/dev/sda: to system OS;
/dev/sdb: to DRBD.
As I said before, you can use only one disk, if leaving one partition exclusive to DRBD.
CTDB:
It's a cluster implementation of TDB, to use with Samba on cluster filesystems. We need it to have active/active cluster, then both the smb services can answers the network requests. More information on http://ctdb.samba.org.
2 Preparing The Nodes
Disabling SELINUX
We need to disable the SELINUX:
vi /etc/selinux/config
Change this line only (leaving everything else untouched):
SELINUX=disabled
Setting Hostname
We need to change the hostname and gateway of the nodes:
vi /etc/sysconfig/network
node1:
NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=node1.clustersmb.int GATEWAY=10.0.0.9
node2:
NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=node2.clustersmb.int GATEWAY=10.0.0.9
Configuring Network Interfaces
Next, we will configure the network interfaces:
node1:
The LAN interface:
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 BOOTPROTO=static IPADDR=10.0.0.181 NETMASK=255.255.255.0 ONBOOT=yes HWADDR=a6:1e:3d:67:66:78
The Cross-Over/DRBD interface:
vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1 BOOTPROTO=static IPADDR=172.16.0.1 NETMASK=255.255.255.0 ONBOOT=yes HWADDR=ee:ef:ff:9a:9a:57
node2:
The LAN interface:
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 BOOTPROTO=static IPADDR=10.0.0.182 NETMASK=255.255.255.0 ONBOOT=yes HWADDR=52:52:a1:1a:62:32
The Cross-Over/DRBD interface:
vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1 BOOTPROTO=static IPADDR=172.16.0.2 NETMASK=255.255.255.0 ONBOOT=yes HWADDR=1a:18:b2:50:96:1e
Setting DNS Configuration
Setting DNS configuration on both nodes (according to your network):
vi /etc/resolv.conf
search clustersmb.int nameserver 10.0.0.9
Configuring Basic Hostname Resolution
Configuring /etc/hosts (same config on both nodes):
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost 10.0.0.191 node1.clustersmb.int node1 10.0.0.192 node2.clustersmb.int node2
PS: You'll probably want to set another lines on this file, to point to other addresses of your network.
Checking Network Connectivity
Let's check if everything is fine:
node1:
Pinging node2 (thru LAN interface):
[root@node1 ~]# ping -c 2 node2
[root@node1 ~]# ping -c 2 node2
PING node2 (10.0.0.182) 56(84) bytes of data.
64 bytes from node2 (10.0.0.182): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from node2 (10.0.0.182): icmp_seq=2 ttl=64 time=0.082 ms
--- node2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.082/0.085/0.089/0.009 ms
Pinging node2 (thru cross-over interface):
[root@node1 ~]# ping -c 2 172.16.0.2
[root@node1 ~]# ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.083 ms
--- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms
node2:
Pinging node1 (thru LAN interface):
[root@node2 ~]# ping -c 2 node1
[root@node2 ~]# ping -c 2 node1
PING node1 (10.0.0.181) 56(84) bytes of data.
64 bytes from node1 (10.0.0.181): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from node1 (10.0.0.181): icmp_seq=2 ttl=64 time=0.063 ms
--- node1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.065/0.068/0.008 ms
Pinging node1 (thru cross-over interface):
[root@node2 ~]# ping -c 2 172.16.0.1
[root@node2 ~]# ping -c 2 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.075 ms
--- 172.16.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.075/0.722/1.369/0.647 ms
Configuring Initialization Options
I like to set runlevel to 3.
vi /etc/inittab
Change this line only (leaving everything else untouched):
id:3:initdefault:
I like remove some services from automatic initialization, to maintain only services that really will be used.
These are the active services that we'll need:
[root@node1 ~]# chkconfig --list | grep 3:sim
[root@node1 ~]# chkconfig --list | grep 3:sim
acpid 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
anacron 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
apmd 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
atd 0:não 1:não 2:não 3:sim 4:sim 5:sim 6:não
cpuspeed 0:não 1:sim 2:sim 3:sim 4:sim 5:sim 6:não
crond 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
irqbalance 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
kudzu 0:não 1:não 2:não 3:sim 4:sim 5:sim 6:não
network 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
rawdevices 0:não 1:não 2:não 3:sim 4:sim 5:sim 6:não
sshd 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
syslog 0:não 1:não 2:sim 3:sim 4:sim 5:sim 6:não
At this point, we need to reboot both nodes to apply configuration.
reboot
