How To Set Up An Active/Passive PostgreSQL Cluster With Pacemaker, Corosync, And DRBD (CentOS 5.5)

This article explains how to set up (and monitor) an Active/Passive PostgreSQL Cluster, using Pacemaker with Corosync and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

Introduction

We use two nodes, one active (that answers requests from apps) and the other on passive mode. If the active server is down, the passive one will automatically take its position (being the active node).

1 Preliminary Note

Linux Distribution:

We are using the CentOS 5.5 (64bits) distribution, but it will probably work on Fedora (and Red Hat, for sure) as well. The installation of the CentOS is very simple and classical, select the base packages and other stuff that you like/need. One issue that must be remembered is that we use DRBD to replicate the PostgreSQL data between the nodes, then you'll need to have a disk or partition exclusive to DRBD. Remember this before partitioning disks on CentOS installation.

Network Hardware/Topology:

We use two Gigabit NIC's per node, one (eth0) connect to the network (LAN), and the other one (eth1) with a cross-over cable connecting both nodes.
The cross-over cable must be used to improve performance and confiability of the system, because DRBD won't depends of network switchs or anything else to replicate data between the nodes.

In this tutorial we will use the physical nodes node1.clusterbr.int and node2.clusterbr.int:
node1.clusterbr.int: Uses IP 10.0.0.191 (LAN) and IP 172.16.0.1 (cross-over)
node2.clusterbr.int: Uses IP 10.0.0.192 (LAN) and IP 172.16.0.2 (cross-over)
dbip.clusterbr.int: It's the Cluster IP, 10.0.0.190. This is the IP that all the applications need to be pointed to, to access the PostgreSQL

Disks:

Both the nodes have two disks:
/dev/sda: to system OS;
/dev/sdb: to DRBD.
As I said before, you can use only one disk, if leaving one partition exclusive to DRBD.

PostgreSQL:

The version of the PostgreSQL used on this article is 8.4, but it doesn't really matters, because anything that you have inside the DRBD device, will be replicated through the cluster.

2 Preparing the Nodes

Disabling SELINUX

We need to disable the SELINUX:

vi /etc/selinux/config

Change this line only (leaving everything else untouched):

SELINUX=disabled

Setting Hostname

We need to change the hostname and gateway of the nodes:

vi /etc/sysconfig/network

node1:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.clusterbr.int
GATEWAY=10.0.0.9

node2:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.clusterbr.int
GATEWAY=10.0.0.9

Configuring network interfaces

Next, we will configure the network interfaces:

node1:

The LAN interface

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.191
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=a6:1e:3d:67:66:78 

The Cross-Over/DRBD interface

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.1
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=ee:ef:ff:9a:9a:57

node2:

The LAN interface

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.192
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=52:52:a1:1a:62:32

The Cross-Over/DRBD interface

vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.2
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=1a:18:b2:50:96:1e

Setting DNS Configuration

Setting DNS configuration on both nodes (according to your network):

vi /etc/resolv.conf

search clusterbr.int
nameserver 10.0.0.9

Configuring basic hostname resolution

Configuring /etc/hosts (same config on both nodes):

vi /etc/hosts

127.0.0.1               localhost.localdomain localhost
10.0.0.191              node1.clusterbr.int   node1
10.0.0.192              node2.clusterbr.int   node2
10.0.0.190              dbip.clusterbr.int   node2

PS: You'll probably want to set another lines on this file, to point to other addresses of your network.

Checking network connectivity

Let's check if everything is fine:

node1:

Pinging node2 (thru LAN interface):

ping -c 2 node2

[root@node1 ~]# ping -c 2 node2
PING node2 (10.0.0.192) 56(84) bytes of data.
64 bytes from node2 (10.0.0.192): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from node2 (10.0.0.192): icmp_seq=2 ttl=64 time=0.082 ms
--- node2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.082/0.085/0.089/0.009 ms

Pinging node2 (thru cross-over interface):

ping -c 2 172.16.0.2

[root@node1 ~]# ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.083 ms
--- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms

node2:

Pinging node1 (thru LAN interface):

ping -c 2 node1

[root@node2 ~]# ping -c 2 node1
PING node1 (10.0.0.191) 56(84) bytes of data.
64 bytes from node1 (10.0.0.191): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from node1 (10.0.0.191): icmp_seq=2 ttl=64 time=0.063 ms
--- node1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.065/0.068/0.008 ms

Pinging node1 (thru cross-over interface):

ping -c 2 172.16.0.1

[root@node2 ~]# ping -c 2 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.075 ms
--- 172.16.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.075/0.722/1.369/0.647 ms

Configuring Initialization options

I like to set runlevel to 3.

vi /etc/inittab

Change this line only (leaving everything else untouched):

id:3:initdefault:

I like remove some services from automatic initialization, to maintain only services that really will be used.

These are the active services that we'll need:

chkconfig --list | grep 3:sim

[root@node1 ~]# chkconfig --list | grep 3:sim
acpid           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
anacron         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
apmd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
atd             0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
cpuspeed        0:não  1:sim   2:sim   3:sim   4:sim   5:sim   6:não
crond           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
irqbalance      0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
kudzu           0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
network         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
rawdevices      0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
sshd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
syslog          0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não

PS: The services that will be managed by the Pacemaker (the Cluster Resource Manager - CRM), in this article they're Postgresql and DRBD, should not been on automatic initialization, because Pacemaker will start/stop these services.

At this point, we need to reboot both nodes to apply configuration.

3. Installing prerequisites and cluster packages

There are some packages that need to be installed before:

yum install -y postgresql84** gcc perl-mailtools perl-dbi php-pgsql

To install the cluster packages, we'll need to add the EPEL repository:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

This link points to the EPEL package to CentOS 5 64bits, be sure to make changes if this is not your distro/version.

Now, we install the ClusterLabs EPEL repository:

wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo

If evething is fine, just go ahead now, and install cluster and drbd packages:

yum install -y pacemaker corosync drbd83 kmod-drbd83 heartbeat