Setting Up An Active/Active Samba CTDB Cluster Using GFS & DRBD (CentOS 5.5)

This article explains how to set up an Active/Active Samba CTDB Cluster, using GFS and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

 

Introduction

We use two nodes, both are active nodes, and the requests are loadbalanced. To replicate data between the nodes we use DRBD. To active/active cluster, we must use a ClusterFS (in this case GFS), to make the two nodes to write on the DRBD resource at same time.

 

1 Preliminary Note

Linux Distribution:

We are using the CentOS 5.5 (64bits) distribution, but will probably work on Fedora (and Red Hat, for sure). The installation of the CentOS is very simple and classical, select the base packages and other stuff that you like/need. One issue that must be remembered is that we use DRBD to replicate data between the nodes, then you'll need to have a disk or partition exclusive to DRBD. Remember this before partitioning disks on CentOS installation.

Network Hardware/Topology:

We use two Gigabit NIC's per node, one (eth0) connect to the network (LAN), and the other one (eth1) with a cross-over cable connecting both nodes.
The cross-over cable must be used to improve performance and confiability of the system, because DRBD won't depends of network switchs or anything else to replicate data between the nodes.

In this tutorial we will use the physical nodes node1.clusterbr.int and node2.clusterbr.int:
node1.clustersmb.int: Uses IP 10.0.0.181 (LAN) and IP 172.16.0.1 (cross-over)
node2.clustersmb.int: Uses IP 10.0.0.182 (LAN) and IP 172.16.0.2 (cross-over)
vip1.clustersmb.int: It's the Cluster IP, from node1 10.0.0.183.
vip2.clustersmb.int: It's the Cluster IP, from node2 10.0.0.184.

Disks:

Both the nodes have two disks:
/dev/sda: to system OS;
/dev/sdb: to DRBD.
As I said before, you can use only one disk, if leaving one partition exclusive to DRBD.

CTDB:

It's a cluster implementation of TDB, to use with Samba on cluster filesystems. We need it to have active/active cluster, then both the smb services can answers the network requests. More information on http://ctdb.samba.org.

 

2 Preparing The Nodes

Disabling SELINUX

We need to disable the SELINUX:

vi /etc/selinux/config

Change this line only (leaving everything else untouched):

SELINUX=disabled

 

Setting Hostname

We need to change the hostname and gateway of the nodes:

vi /etc/sysconfig/network

node1:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.clustersmb.int
GATEWAY=10.0.0.9

node2:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.clustersmb.int
GATEWAY=10.0.0.9

 

Configuring Network Interfaces

Next, we will configure the network interfaces:

node1:

The LAN interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.181
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=a6:1e:3d:67:66:78

The Cross-Over/DRBD interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.1
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=ee:ef:ff:9a:9a:57

node2:

The LAN interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.182
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=52:52:a1:1a:62:32

The Cross-Over/DRBD interface:

vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.2
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=1a:18:b2:50:96:1e

 

Setting DNS Configuration

Setting DNS configuration on both nodes (according to your network):

vi /etc/resolv.conf
search clustersmb.int
nameserver 10.0.0.9

 

Configuring Basic Hostname Resolution

Configuring /etc/hosts (same config on both nodes):

vi /etc/hosts
127.0.0.1               localhost.localdomain localhost
10.0.0.191              node1.clustersmb.int   node1
10.0.0.192              node2.clustersmb.int   node2

PS: You'll probably want to set another lines on this file, to point to other addresses of your network.

 

Checking Network Connectivity

Let's check if everything is fine:

node1:

Pinging node2 (thru LAN interface): 

[[email protected] ~]# ping -c 2 node2

[[email protected] ~]# ping -c 2 node2
PING node2 (10.0.0.182) 56(84) bytes of data.
64 bytes from node2 (10.0.0.182): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from node2 (10.0.0.182): icmp_seq=2 ttl=64 time=0.082 ms
--- node2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.082/0.085/0.089/0.009 ms

Pinging node2 (thru cross-over interface):

[[email protected] ~]# ping -c 2 172.16.0.2

[[email protected] ~]# ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.083 ms
--- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms

node2:

Pinging node1 (thru LAN interface): 

[[email protected] ~]# ping -c 2 node1

[[email protected] ~]# ping -c 2 node1
PING node1 (10.0.0.181) 56(84) bytes of data.
64 bytes from node1 (10.0.0.181): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from node1 (10.0.0.181): icmp_seq=2 ttl=64 time=0.063 ms
--- node1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.065/0.068/0.008 ms

Pinging node1 (thru cross-over interface):

[[email protected] ~]# ping -c 2 172.16.0.1

[[email protected] ~]# ping -c 2 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.075 ms
--- 172.16.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.075/0.722/1.369/0.647 ms

 

Configuring Initialization Options

I like to set runlevel to 3.

vi /etc/inittab

Change this line only (leaving everything else untouched):

id:3:initdefault:

I like remove some services from automatic initialization, to maintain only services that really will be used.

These are the active services that we'll need:

[[email protected] ~]# chkconfig --list | grep 3:sim

[[email protected] ~]# chkconfig --list | grep 3:sim
acpid           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
anacron         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
apmd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
atd             0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
cpuspeed        0:não  1:sim   2:sim   3:sim   4:sim   5:sim   6:não
crond           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
irqbalance      0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
kudzu           0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
network         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
rawdevices      0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
sshd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
syslog          0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não

At this point, we need to reboot both nodes to apply configuration.

reboot
Share this page:

Suggested articles

1 Comment(s)

Add comment

Comments

By: pb
Reply  

WTF?! why did you disable SELINUX?? please read /etc/samba/smb.conf for info on how to make samba play nicely. NEVER disable SELINUX, doing so is like chmod 777, yes you do make it work but... While getting your head around contexts and such may I recommend SELINUX=permissive so you can at some point re-enable it to harden your server. If you tried to do a full system relabel now you will most likely get some MASSIVE issues.

getenforce (is the system enforcing or permissive?)
setenforce 0 (set SELINUX on the fly to permissive)
setenforce 1 (set SELINUX on the fly to enforcing)
# the system will default back to what ever is defined in /etc/selinux/config upon reboot. this will NOT work if you have at any point disabled SELINUX

From Fedora's smb.conf:
chcon -R -t samba_share_t /path/to/share

all newly created files will automatically inherit the context of their parent.

you will not find the samba_share_t context listed anywhere other than smb.conf as there is no default system location for a samba share, unlike the default httpd doc root. If you were running an httpd server you can find out the needed contexts by simply:
semanage fcontext -l|grep httpd


:D