Configuring A High Availability Cluster (Heartbeat) On CentOS

Want to support HowtoForge? Become a subscriber!
 
Submitted by babarsaleem (Contact Author) (Forums) on Tue, 2007-10-30 11:30. :: CentOS

Configuring A High Availability Cluster (Heartbeat) On CentOS

This guide shows how you can set up a two node, high-availability HTTP cluster with heartbeat on CentOS. Both nodes use the Apache web server to serve the same content.

 

Pre-Configuration Requirements

  1. Assign hostname node01 to primary node with IP address 172.16.4.80 to eth0.
  2. Assign hostname node02 to slave node with IP address 172.16.4.81.

Note: on node01

uname -n

must return node01.

On node02

uname -n

must return node02.

172.16.4.82 is the virtual IP address that will be used for our Apache webserver (i.e., Apache will listen on that address).

 

Configuration

1. Download and install the heartbeat package. In our case we are using CentOS so we will install heartbeat with yum:

yum install heartbeat

or download these packages:

heartbeat-2.08
heartbeat-pils-2.08
heartbeat-stonith-2.08

2. Now we have to configure heartbeat on our two node cluster. We will deal with three files. These are:

authkeys
ha.cf
haresources

3. Now moving to our configuration. But there is one more thing to do, that is to copy these files to the /etc/ha.d directory. In our case we copy these files as given below:

cp /usr/share/doc/heartbeat-2.1.2/authkeys /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.2/ha.cf /etc/ha.d/
cp /usr/share/doc/heartbeat-2.1.2/haresources /etc/ha.d/

4. Now let's start configuring heartbeat. First we will deal with the authkeys file, we will use authentication method 2 (sha1). For this we will make changes in the authkeys file as below.

vi /etc/ha.d/authkeys

Then add the following lines:

auth 2
2 sha1 test-ha

Change the permission of the authkeys file:

chmod 600 /etc/ha.d/authkeys

5. Moving to our second file (ha.cf) which is the most important. So edit the ha.cf file with vi:

vi /etc/ha.d/ha.cf

Add the following lines in the ha.cf file:

logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
bcast eth0
udpport 694
auto_failback on
node node01
node node02

Note: node01 and node02 is the output generated by

uname -n

6. The final piece of work in our configuration is to edit the haresources file. This file contains the information about resources which we want to highly enable. In our case we want the webserver (httpd) highly available:

vi /etc/ha.d/haresources

Add the following line:

node01 172.16.4.82 httpd

7. Copy the /etc/ha.d/ directory from node01 to node02:

scp -r /etc/ha.d/ root@node02:/etc/

8. As we want httpd highly enabled let's start configuring httpd:

vi /etc/httpd/conf/httpd.conf

Add this line in httpd.conf:

Listen 172.16.4.82:80

9. Copy the /etc/httpd/conf/httpd.conf file to node02:

scp /etc/httpd/conf/httpd.conf root@node02:/etc/httpd/conf/

10. Create the file index.html on both nodes (node01 & node02):

On node01:

echo "node01 apache test server" > /var/www/html/index.html

On node02:

echo "node02 apache test server" > /var/www/html/index.html

11. Now start heartbeat on the primary node01 and slave node02:

/etc/init.d/heartbeat start

12. Open web-browser and type in the URL:

http://172.16.4.82

It will show node01 apache test server.

13. Now stop the hearbeat daemon on node01:

/etc/init.d/heartbeat stop

In your browser type in the URL http://172.16.4.82 and press enter.

It will show node02 apache test server.

14. We don't need to create a virtual network interface and assign an IP address (172.16.4.82) to it. Heartbeat will do this for you, and start the service (httpd) itself. So don't worry about this.

Don't use the IP addresses 172.16.4.80 and 172.16.4.81 for services. These addresses are used by heartbeat for communication between node01 and node02. When any of them will be used for services/resources, it will disturb hearbeat and will not work. Be carefull!!!


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Sergei Petrunin (not registered) on Tue, 2012-06-26 21:36.
It exactly worked for me on two virtual boxes with CentOS6.2!

This possobility to stay service on line even is node is down is awesome!

I'm a noob in field of "HA" or "Clustering"... And I have little question:

What similar projects to "Heartbeat" you know? Where is the difference?

e.g.: What is the relationship between "Heatbeat" project and "Apache Hadoop" project? Is "Heartbeat" is like a part of "Hadoop" or it is something completely different?

 

 Sergi

Submitted by Avinash (not registered) on Sat, 2012-03-31 10:10.

getting error as below:

heartbeat[7364]: 2012/03/26_15:41:25 info: Version 2 support: false
heartbeat[7364]: 2012/03/26_15:41:25 ERROR: No heartbeat media defined
heartbeat[7364]: 2012/03/26_15:41:25 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[7364]: 2012/03/26_15:41:25 ERROR: Configuration error, heartbeat not started.

my authkeys file:

auth 2
2 sha1 test1-HA

my haresources file:

server1 192.168.225.134 httpd

my ha.cf file:

node server1
node server2

Submitted by joker400 (not registered) on Sat, 2012-02-18 10:30.

Thanks for the great tutorial, it is really helped me 


Submitted by Nitin ray (not registered) on Fri, 2011-07-29 06:55.
Thanks man, it's really helpful to us.
Submitted by tundasrl (registered user) on Sun, 2011-06-12 18:01.

that didn't work for me until I found another user who solved adding the netmask size in harerources.
In my case I used a "haresources" like this:

node01 192.168.1.92/24 httpd

Submitted by tquang (registered user) on Tue, 2010-12-07 04:51.

What happened if had 1 IP WAN in this configuration?

Node in Local:
Node1: 192.168.0.1/24
Node2: 192.168.0.2/24

And IP WAN:
ISP issue: 208.67.222.222/20

Submitted by Anonymous (not registered) on Mon, 2010-11-29 01:38.
I've got better results on CentOS with the Gridlock high availability cluster, see Gridlock. Yes, it's not free but I've just had it with wasting time with open source.
Submitted by Anonymous (not registered) on Sat, 2011-05-28 18:43.

It actually doesnot work better with Gridlock, it is just that you have no clue what you are doing.

Try some basic linux tuts before commenting here

Submitted by Sushant Chawla (not registered) on Mon, 2010-09-20 12:01.

Hi

First of all thanks for this very easy to understand tutorial.

My problem is that my httpd server is not running if I am binding it on virtual ip address, it shows the error:

(99)Cannot assign requested address: make_sock: could not bind to address 192.168.15.250:80
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]

 

If I enable httpd to listen on all interface heartbeat is running as expected. Please correct me if I am wrong anywhere. I have checked your howto 3-4 times & followed exactly what you have stated.

 Please help...

Regds

Sushant Chawla

Submitted by shahji (registered user) on Fri, 2010-11-26 14:19.

HI

I have properly installed and configured Heartbeat (heartbeat-2.1.3-3.el5.centos) on CentOS release 5.4. I have question as I stop heartbeat on node01 it automatically shift to node02. If i stop node02 it should automatically shift to node01 but it doesnot.
I need help regarding this(As I stop Heartbeat on Node01 it automatically shift to Node02 but vice versa it does not work, means if i stop heartbeat on Node02 it should failover to Node01, how can I do this.

 

Regards
SHAH JI

Submitted by ravinder (not registered) on Sun, 2012-12-23 06:19.

In ha.cf file auto_failback should be on

auto_failback on

The master listed in the haresources file holds all the resources until a failover, at which time the slave takes over. When auto_failback is set to on once the master comes back online, it will take everything back from the slave. When set to off this option will prevent the master node from re-acquiring cluster resources after a failover.

Submitted by Arn (not registered) on Fri, 2010-09-10 07:47.

Thanks for providing such a nice configuration doc.

As per the doc I did the configuration and it is working fine. But when i stopped httpd service on node01 HA is not able to switch to node2's httpd.

Actions taken :

step 1: Heartbeat is working fine as per your doc

step 2: stopped the httpd service on  node1

step 3:  http://172.16.4.82

            Which is giving a black page. Why ?

 -----------------------------------------------------------------------------------

 

Question 2:- How to configure multiple service on Heartbeat

      service for httpd and  mysql together. If any of the service failed it show be able to switch to next server. 

Submitted by Anonymous (not registered) on Fri, 2010-08-06 08:32.

port on

172.16.4.82:80
 redirects to
172.16.4.81:80
IE. 82 > 81
 That is expected .

BUT
can ping the powered off box (unexpected)

Any Reason

Submitted by Kensai Yanesha (not registered) on Tue, 2010-04-06 11:08.
I also want to thank you for this great tutorial, everything worked!
Submitted by paul (not registered) on Wed, 2009-11-11 02:53.

Hi,
 How can I configure heartbeat so that it will pass control to second node if apache in first node fails?
 When I shut down node1 it switch control to node2. But if apache (only apche not heartbeat) stopped in first node, it will not pass control to second node.
 
 Paul
 

Submitted by Tony (not registered) on Wed, 2009-11-04 21:58.

One thing not mentioned here and something that may save someone a few hours is to make sure that there is no Firewall configured that might be blocking UDP on port 694.

When this occurs it is not apparent what is going wrong and the the HA setup immediately goes into a split brain scenario. Its quite frustrating as your installation and configuration will be correct.

A comment that I would make on the above is that for an active/active setup there is no need to configure specific resources such as httpd. This has the benefit of allowing the reaction times to be much quicker and for having services such as httpd and mysqld etc be configured for startup on reboot, irrespective of the actions of heartbeat.

Tony

 

Submitted by chalitha (not registered) on Fri, 2009-10-30 17:57.
thanks man . keep it up. great work.
Submitted by Thusith (not registered) on Thu, 2009-08-06 14:40.

I tried with CentOS 5.3 and it works!!! Thank You.

Submitted by rootlurker (not registered) on Tue, 2009-06-02 09:20.

Good job! Great howto's.. it's works very well.

 Many thanks...

Submitted by Carlos (not registered) on Tue, 2009-01-20 19:48.

Thanks for the tutorial, it worked really well.

Good Job.

Submitted by Jesser (not registered) on Wed, 2009-01-07 20:09.

Excelent How-to! Works fine for me!

CentOS 5.2 x86 default install with heartbeat-2.1.3-3.el5.

Thanks!

 

Submitted by Webmaster (not registered) on Fri, 2008-12-19 06:22.

Hi There,

 Everything is working but only starting HTTPd Service. I am using OpenVZ to simulate these HA. But on each node, it doesn't have eth0 interface. only venet0:0

 Any ideas?

[root@node01 ~]# netstat -untap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      28089/sendmail: acc
tcp        0      0 :::22                       :::*                        LISTEN      27998/sshd          
tcp        0      0 ::ffff:192.168.150.80:22    ::ffff:192.168.150.1:55656  ESTABLISHED 1673/0              

[root@node01 ~]# service httpd start
Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address 192.168.150.82:80
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]
[root@node01 ~]#

Submitted by Deepak. (not registered) on Tue, 2010-05-18 07:40.

hye.!

This is a problem which may arise sometime if your virtual IP is not configured properly.

First of all ensure that you have configured virtual IP on both of ur nodes primary and secondary. here 192.168.150.80 must be the IP adder of your primary node and 192.168.0150.82 be the virtual IP.

most probably this will solve your problem.

 

 

Submitted by chr1x2 (not registered) on Wed, 2009-07-29 07:43.

do you have "which" command?

 # rpm -qa | grep which

Maybe heartbeat cannot find the gawk command. Heartbeat need the which command to find those commands on your system.

Submitted by DrLove73 (not registered) on Thu, 2009-01-01 17:53.

Go to /etc/ha.d/ha.cf and use/change

"bcast eth0 eth1" to

"bcast venet0 venet1" etc.

Do not use venet0:0 or similar since heartbeat is using interfaces not IP addresses.

Submitted by lakhera2009 (registered user) on Wed, 2009-06-10 02:39.

I am facing the same issue.I try to change

bcast venet0 

Bur again the same error 

Heartbeat is running fine but it was not able to bring up the resource when i manually try to bring up the resource

 

 service httpd start
Starting httpd: (99)Cannot assign requested address: make_sock: could not bind to address 192.168.1.113:80
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]

Any help is highly appreciated

Submitted by Anonymous (not registered) on Fri, 2008-12-12 14:59.
Don't follow this manual, its confusing and sucks.
Submitted by Anonymous (not registered) on Sat, 2013-08-17 22:16.

Why say do not follow this Post, it is useful but is true missing something for Version 2, I just want to make a note for all your guy for Ref, in version 2 if you enabled "crm on" then Heartbeat will not use the "haresources" file any more and the service/virtual IP address could not create. In fact it is playing me whole night to Google search and no page saying this until I find "http://www.netexpertise.eu/en/linux/heartbeat-2-howto.html/comment-page-1#comment-15393"

Submitted by Anonymous (not registered) on Wed, 2009-10-21 14:46.

Respected Sir/Mam,

From last few days, I am working to implement High Availability Cluster with apache and heartbeat.

Resources I am using are as follow:

CentOS-5  as master-node
fedora-11 as slave-node

hearbeat  2.1.4
apache 2.2

I referred tutorial : http://www.howtoforge.com/high_availability_heartbeat_centos

At the moment, I have successfully implemented heartbeat but the problem is with apache httpd service which is not automatically getting started on heartbeat start up.

So, please guide me regarding this issue.
It would be of great help.

Reply me on niraj874u@gmail.com

Submitted by babarsaleem (registered user) on Tue, 2008-07-22 12:33.
1st accept my appoligies for replying  late. and then let me thank you for appreciating this work. Both statements have same meanings to hearbeat, perhaps i don't know why error generated.
Submitted by limbo (registered user) on Wed, 2008-04-30 23:22.

Firstly, this was a great tutorial, thanks. I had trouble starting the heartbeat service using this node directive syntax in the ha.cf file

node node01
node node02

this however worked for me

node node01 node02

I am using centos 5.1 and heartbeat v2.1.3 (this might have changed since v2.1.2??) and more info on the node directive is here: http://linux-ha.org/ha.cf/NodeDirective

Submitted by raducu (not registered) on Fri, 2009-01-09 13:27.
This article is really great. I followed the instruction and everything worked smoothly on centos 5.2/ Thank you very much.
Submitted by Adam (not registered) on Thu, 2009-01-22 17:53.

the haresources file has been replaced with a XML-based cluster-wide configuration file in heartbeat v2.

 Take a look here how to properly setup heartbeat.

http://www.linux-ha.org/v2/Examples/Simple

Submitted by Krishna (not registered) on Sat, 2009-05-02 13:37.

Dear All,

I have done exactly same entries in my ha.cf file, but when I run uname -n command it does not show me both of the nodes ? Any idea how to correct it. Please help ?

Submitted by sunny (not registered) on Tue, 2011-11-29 15:53.

Did you got any solution for this problem, Even I am facing same issue.

 

Thanks

Submitted by cuky23 (not registered) on Tue, 2013-06-18 22:33.

I have got this setup so that the resource group just contains IP/MASK no services, as I want the services to be running on both nodes.

I am using haproxy to route traffic to both servers from the VIP's .  managed to get VIP running on each server one for WEB and one for SQL.

Only issue I have found is that if both nodes go off.  It can be a bit of pain to the service heartbeat start to kick in again.  in fact I have had to get the second node started before it will start heartbeat on the first node?  Anyone explain why happens.