How To Set Up An Active/Passive PostgreSQL Cluster With Pacemaker, Corosync, And DRBD (CentOS 5.5)

This article explains how to set up (and monitor) an Active/Passive PostgreSQL Cluster, using Pacemaker with Corosync and DRBD. Prepared by Rafael Marangoni, from BRLink Servidor Linux Team.

 

Introduction

We use two nodes, one active (that answers requests from apps) and the other on passive mode. If the active server is down, the passive one will automatically take its position (being the active node).

 

1 Preliminary Note

Linux Distribution:

We are using the CentOS 5.5 (64bits) distribution, but it will probably work on Fedora (and Red Hat, for sure) as well. The installation of the CentOS is very simple and classical, select the base packages and other stuff that you like/need. One issue that must be remembered is that we use DRBD to replicate the PostgreSQL data between the nodes, then you'll need to have a disk or partition exclusive to DRBD. Remember this before partitioning disks on CentOS installation.

Network Hardware/Topology:

We use two Gigabit NIC's per node, one (eth0) connect to the network (LAN), and the other one (eth1) with a cross-over cable connecting both nodes.
The cross-over cable must be used to improve performance and confiability of the system, because DRBD won't depends of network switchs or anything else to replicate data between the nodes.

In this tutorial we will use the physical nodes node1.clusterbr.int and node2.clusterbr.int:
node1.clusterbr.int: Uses IP 10.0.0.191 (LAN) and IP 172.16.0.1 (cross-over)
node2.clusterbr.int: Uses IP 10.0.0.192 (LAN) and IP 172.16.0.2 (cross-over)
dbip.clusterbr.int: It's the Cluster IP, 10.0.0.190. This is the IP that all the applications need to be pointed to, to access the PostgreSQL

Disks:

Both the nodes have two disks:
/dev/sda: to system OS;
/dev/sdb: to DRBD.
As I said before, you can use only one disk, if leaving one partition exclusive to DRBD.

PostgreSQL:

The version of the PostgreSQL used on this article is 8.4, but it doesn't really matters, because anything that you have inside the DRBD device, will be replicated through the cluster.

 

2 Preparing the Nodes

Disabling SELINUX

We need to disable the SELINUX:

vi /etc/selinux/config

Change this line only (leaving everything else untouched):

SELINUX=disabled

 

Setting Hostname

We need to change the hostname and gateway of the nodes:

vi /etc/sysconfig/network

node1:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node1.clusterbr.int
GATEWAY=10.0.0.9

node2:

NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=node2.clusterbr.int
GATEWAY=10.0.0.9

 

Configuring network interfaces

Next, we will configure the network interfaces:

node1:

The LAN interface

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.191
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=a6:1e:3d:67:66:78

The Cross-Over/DRBD interface

vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.1
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=ee:ef:ff:9a:9a:57

node2:

The LAN interface

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.0.0.192
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=52:52:a1:1a:62:32

The Cross-Over/DRBD interface

vi /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=172.16.0.2
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=1a:18:b2:50:96:1e

 

Setting DNS Configuration

Setting DNS configuration on both nodes (according to your network):

vi /etc/resolv.conf
search clusterbr.int
nameserver 10.0.0.9

 

Configuring basic hostname resolution

Configuring /etc/hosts (same config on both nodes):

vi /etc/hosts
127.0.0.1               localhost.localdomain localhost
10.0.0.191              node1.clusterbr.int   node1
10.0.0.192              node2.clusterbr.int   node2
10.0.0.190              dbip.clusterbr.int   node2

PS: You'll probably want to set another lines on this file, to point to other addresses of your network.

 

Checking network connectivity

Let's check if everything is fine:

node1:

Pinging node2 (thru LAN interface):

ping -c 2 node2


[[email protected] ~]# ping -c 2 node2
PING node2 (10.0.0.192) 56(84) bytes of data.
64 bytes from node2 (10.0.0.192): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from node2 (10.0.0.192): icmp_seq=2 ttl=64 time=0.082 ms
--- node2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.082/0.085/0.089/0.009 ms

Pinging node2 (thru cross-over interface):

ping -c 2 172.16.0.2

[[email protected] ~]# ping -c 2 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.083 ms
--- 172.16.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.083/0.083/0.083/0.000 ms

node2:

Pinging node1 (thru LAN interface):

ping -c 2 node1

[[email protected] ~]# ping -c 2 node1
PING node1 (10.0.0.191) 56(84) bytes of data.
64 bytes from node1 (10.0.0.191): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from node1 (10.0.0.191): icmp_seq=2 ttl=64 time=0.063 ms
--- node1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.063/0.065/0.068/0.008 ms

Pinging node1 (thru cross-over interface):

ping -c 2 172.16.0.1

[[email protected] ~]# ping -c 2 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 172.16.0.1: icmp_seq=2 ttl=64 time=0.075 ms
--- 172.16.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.075/0.722/1.369/0.647 ms

 

Configuring Initialization options

I like to set runlevel to 3.

vi /etc/inittab

Change this line only (leaving everything else untouched):

id:3:initdefault:

I like remove some services from automatic initialization, to maintain only services that really will be used.

These are the active services that we'll need:

chkconfig --list | grep 3:sim

[[email protected] ~]# chkconfig --list | grep 3:sim
acpid           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
anacron         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
apmd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
atd             0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
cpuspeed        0:não  1:sim   2:sim   3:sim   4:sim   5:sim   6:não
crond           0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
irqbalance      0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
kudzu           0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
network         0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
rawdevices      0:não  1:não  2:não  3:sim   4:sim   5:sim   6:não
sshd            0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não
syslog          0:não  1:não  2:sim   3:sim   4:sim   5:sim   6:não

PS: The services that will be managed by the Pacemaker (the Cluster Resource Manager - CRM), in this article they're Postgresql and DRBD, should not been on automatic initialization, because Pacemaker will start/stop these services.

At this point, we need to reboot both nodes to apply configuration.

 

3. Installing prerequisites and cluster packages

There are some packages that need to be installed before:

yum install -y postgresql84** gcc perl-mailtools perl-dbi php-pgsql

To install the cluster packages, we'll need to add the EPEL repository:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

This link points to the EPEL package to CentOS 5 64bits, be sure to make changes if this is not your distro/version.

Now, we install the ClusterLabs EPEL repository:

wget -O /etc/yum.repos.d/pacemaker.repo http://clusterlabs.org/rpm/epel-5/clusterlabs.repo

If evething is fine, just go ahead now, and install cluster and drbd packages:

yum install -y pacemaker corosync drbd83 kmod-drbd83 heartbeat
Share this page:

5 Comment(s)

Add comment

Please register in our forum first to comment.

Comments

By:
Reply  

Very nice and happily confirms that my solution (although using Debian Lenny) was spot on.

Also very cool to see some additional tools mentioned (zabbix f.i.)

Thanks for the efforts!

 

One word though to anyone using the HA stuff, there are some versions that can be 'complicated' so watch out and if possible, use known working versions.

By: Serge Dubrouski
Reply  

Nice work!

Could you place it to the Clusterlabs' HowTo page? There is a basic HowTo for PostgreSQL that I created earlier, it would be good to replace it with yours.

By: Gergely Polonkai
Reply  

Dear everyone, 

please stop disabling SELinux. It is good, and works more than fine. If you really have trouble with it, set it to permissive mode, and create a custom house rule for that. Most importantly, if you use packages from the central CentOS/Fedora/RedHad repository, report these troubles so they can add the required changes to their policies.

It's a nice article, though, it goes through every necessary steps, thank you!

Best wishes!

Gergely

By: FattahRozzaq
Reply  

Very nice tutorial. I use your article to help me build PGSQL cluster using Postgres-XC (PGXC).

Would you show me to use the crm with PGXC?

(Because I don't use the standard PostgreSQL server, I use Postgres-XC).

 

Thank you in advance!

FattahRozzaq

By: Jasmin
Reply  

Thank you so much for this great detailed tutorial.

I'm having toruble with configuring resources for cluster. Since I'm on RHEL 6.7 there are newer versions of pacemaker and corosync and now configuring is done with pcs instead of crm. It has different syntax.

Is there any possibility someone can help with that.

i.e.

how would this command go with pcs

crm configure ms ms_drbd_postgres drbd_postgres \  meta master-max="1" master-node-max="1" \  clone-max="2" clone-node-max="1" \  notify="true"