LB1 Not Taking Over Shared IP Using Heartbeat 2.1.3 On FC10 64Bit

[asyadiqin]

I have a freshly installed FC10 64Bit server on which I installed HAProxy & Heartbeat using the howto tutorial in this site.

Setting Up A High-Availability Load Balancer (With Failover and Session Support) With HAProxy/Heartbeat On Fedora 8 (http://www.howtoforge.com/high-availability-load-balancer-haproxy-heartbeat-fedora8)

There was no error when installing the packages using yum. I started Heartbeat on the server, ie. lb1.
[root@lb1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services:
2009/01/21_22:30:05 INFO: Resource is stopped
Done.

As per section 3.4.4, I run the following command
ip addr sh eth0

Instead of the output specified in the howto, this is the output I get
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:15:c5:5f:d4:78 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.3/24 brd 10.1.1.255 scope global eth0
inet6 fe80::215:c5ff:fe5f:d478/64 scope link
valid_lft forever preferred_lft forever

As you can see above, the LB1 is not binding to the shared IP at all. I stopped Heartbeat and run the following command
sh /usr/lib64/heartbeat/BasicSanityCheck

Below is the output
Using interface: eth0
Starting base64 and md5 algorithm tests
base64 and md5 algorithm tests succeeded.
Starting Resource Agent tests
Testing RA: Dummy
Testing RA: IPaddr
Testing RA: IPaddr2
Testing RA: Filesystem
RA tests PASSED
Starting IPC tests
Starting heartbeat
Starting High-Availability services:
2009/01/21_22:09:33 INFO: Resource is stopped
Done.

Reloading heartbeat
Reloading heartbeat
Stopping heartbeat
Stopping High-Availability services:
Done.

Checking STONITH basic sanity.
Performing apphbd success case tests
Performing apphbd failure case tests
Starting LRM tests
Starting heartbeat
Starting High-Availability services:
2009/01/21_22:10:48 INFO: Resource is stopped
Done.

starting STONITH Daemon tests
STONITH Daemon tests passed.
Stopping heartbeat
Stopping High-Availability services:
Done.

Starting CRM tests
CRM tests passed.
heartbeat[26273]: 2009/01/21_22:10:02 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:02 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:02 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:02 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:02 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:02 ERROR: write_child: write failure on ping 127.0.0.1.: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:03 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:03 ERROR: write_child: write failure on ping 127.0.0.1.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:03 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:03 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:03 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:03 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping 127.0.0.1.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping 127.0.0.1.: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26271]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping 127.0.0.1.: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[26273]: 2009/01/21_22:10:04 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
heartbeat[29128]: 2009/01/21_22:11:06 ERROR: glib: Error sending packet: Operation not permitted
heartbeat[29128]: 2009/01/21_22:11:06 ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
OOPS! Looks like we had some errors come up.
1 errors. Log file is stored in /tmp/linux-ha.testlog

Below is the part of the content of the log file in /tmp/linux-ha.testlog
Jan 21 22:12:08 lb1.iknow-uk.com CTS: debug: Cluster is inactive
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:05 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: glib: Error sending packet: Operation not permitted
Jan 21 22:12:08 lb1.iknow-uk.com CTS: BadNews: Jan 21 22:12:06 lb1.iknow-uk.com heartbeat: [29639]: ERROR: write_child: write failure on ping_group pingus_please.: Operation not permitted

Anyone have any idea what I did wrong? FYI, LB1 is running Fedora Core 10 64Bit. Any help or assistance is appreciated.

[falko]

Is SELinux disabled?
Are there any other errors in syslog?

[asyadiqin]

Is SELinux disabled?
Are there any other errors in syslog?

I had disabled both the firewall and Selinux during installation as I don't actually need it as I have a dedicated firewall server.

As for other errors in syslog, which log file should I be looking at? Should it be in /var/log? I did enabled debugging in heartbeat and its writing to /var/log/ha-debug. Is this what you are referring to?

[falko]

I'd check out all logs in the /var/log directory.

[adam0x54]

check your bcast configuration in ha.cf in heartbeat. try commenting out bcast eth0 and mcast and leave ucast eth0 ip-of-other-machine and do the same for the other node. Go to linux-ha.org and try to figure out hearbeat v2. Its a pita to set it up, i am working on it.

-Adam

[asyadiqin]

check your bcast configuration in ha.cf in heartbeat. try commenting out bcast eth0 and mcast and leave ucast eth0 ip-of-other-machine and do the same for the other node. Go to linux-ha.org and try to figure out hearbeat v2. Its a pita to set it up, i am working on it.

-Adam

FIXED! All that I needed was to add the CIDR in haresource.

lb1.iknow-uk.com 10.1.1.2
to
lb1.iknow-uk.com 10.1.1.2/24

Anyway, its working fine as it is now. Thanks for everyone help.