Efficient High-Available LoadBalanced Cluster On CentOS 5.3 (Direct Routing Method)

Want to support HowtoForge? Become a subscriber!
 
Submitted by hediehamirjahanshahi (Contact Author) (Forums) on Mon, 2010-02-15 17:47. :: CentOS | High-Availability

Efficient High-Available LoadBalanced Cluster On CentOS 5.3 (Direct Routing Method)

This article explains how to set up an LVS cluster of load balanced virtual servers with Heartbeat and Ldirectord On CentOS 5.3.The load balancer sits between the user and two (or more) backend Apache/IIS web servers that hold the same content. Not only does the load balancer distribute the requests to the two backend Apache/IIS servers, it also checks the health of the backend servers. If one of them is down, all requests will automatically be redirected to the remaining backend server.

 

Introduction

An LVS cluster consists or one or more virtual services each of which may have zero or more real servers. The IP address of a virtual service is what end-users connect to and is typically advertised over DNS. When a connection is made to a virtual service, it is allocated a real server, and all packets for this connection are forwarded to this real server. Ldirectord is a daemon to monitor and administer real servers in a LVS cluster of load balanced virtual servers. Ldirectord typically used as a resource for Linux-HA. Ldirectord monitors the health of the real servers by periodically requesting a known URL and checking that the response contains an expected response. If a real server fails then the server is removed and will be reactivated once it comes back on line. If all the real servers are down then a fall-back server is inserted into the pool, which will made quiescent one of the real web servers comes back on line. Typically, the fall-back server is localhost. If an HTTP virtual service is being provided then it is useful to run an Apache HTTP server that returns a page indicating that the service is temporarily inaccessible.

Note: This tutorial is based on my personal experience and some other tutorials which is publicly available on Internet. I do not issue any guarantee that this will work for you!.

 

Preliminary Note

In this tutorial I will use the following 3 hosts:
Virtual IP address (end users connect to this) : 10.10.10.53
Load Balancer: ld.example.com, IP address: 10.10.10.52
Web Server 1: http1.example.com, IP address: 192.168.200.102
Web Server 2: http2.example.com, IP address: 192.168.200.103

 

Load Balancer Configuration

Install heartbeat,heartbeat-ldirector and ipvsadm packages on your Load Balancer system (ld.example.com).

yum install heartbeat heartbeat-ldirector ipvsadm -y
chkconfig ldirectord off
chkconfig heartbeat on
sed -i 's/net.ipv4.ip_forward = 1/net.ipv4.ip_forward = 0' /etc/sysctl.conf
sysctl -p

 

Load Balancer Secondary Ethernet Configuration

Configure secondary eth0 for LVS as its going to be exposed to outside world or your local gateway.

vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0:0
BOOTPROTO=none
ONBOOT=yes
HWADDR=3a:5d:71:ad:67:47
NETMASK=255.255.255.0
IPADDR=10.10.10.52
GATEWAY=10.10.10.1
TYPE=Ethernet

vi /etc/sysconfig/network-scripts/ifcfg-eth0:0

DEVICE=eth0:0
BOOTPROTO=none
ONBOOT=yes
HWADDR=3a:5d:71:ad:67:47
NETMASK=255.255.255.0
IPADDR=10.10.10.53
TYPE=Ethernet

service network restart

 

Configuring ldirectord

Configure ldirectord on your Load Balancer system.

vi /etc/ha.d/ldirectord.cf

checktimeout=30
checkinterval=2
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=no
virtual=10.10.10.53:80
        fallback=127.0.0.1:80
        real=192.168.200.102:80 gate
        real=192.168.200.103:80 gate
        service=http
        request="/check.txt"
        httpmethod=GET
        receive="webserverisworking"
        persistent=100
        scheduler=lblc
        protocol=tcp
        checktype=negotiate

Important Note: after virtual=x.x.x.x:80 line , each line MUST start with TAB. Don't forget to press TAB key before each lines.

service ldirectord start

In the virtual= line we put our virtual IP address (10.10.10.53 in this example), and in the real= lines we list the IP addresses of our Apache/IIS nodes (192.168.200.102 and 192.168.200.103 in this example). In the request= line we list the name of a file on http1 and http2 that ldirectord will request repeatedly to see if http1 and http2 are still alive. That file (that we are going to create later on) must contain the string listed in the receive= line.
In the scheduler= line you can use one of the following method depending on your needs: rr - wrr - lc - wlc - lblc - lblcr - dh - sh - sed - nq

For more information about scheduler methods visit: http://linux.die.net/man/8/ipvsadm

 

Configure heartbeat

Configure heartbeat on your Load Balancer system.

vi /etc/ha.d/ha.cf

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility     local0
keepalive 2
deadtime 10
bcast   eth0
mcast eth0 225.0.0.1 694 1 0
auto_failback on
respawn hacluster /usr/lib/heartbeat/ipfail
node ld.example.com

Important: As nodenames we must use the output of:

uname -n

vi /etc/ha.d/haresources

ld.example.com ldirectord::ldirectord.cf LVSSyncDaemonSwap::master IPaddr2::10.10.10.53/24/eth0/10.10.10.255

The first word in the first line above is the output of

uname -n

vi /etc/ha.d/authkeys

auth 3
3 md5 somerandomstring

chmod 600 /etc/ha.d/authkeys

 

Testing

Let's check if load balancer work as expected:

ip addr sh eth0

The load balancer should list the virtual IP address (10.10.10.53):

2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:c8:6f:2f brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.52/24 brd 10.10.10.255 scope global eth0
    inet 10.10.10.53/24 brd 10.10.10.255 scope global secondary eth0:0

If your tests went fine, you can now go on and configure the two Apache/IIS nodes.

 

Cluster Nodes Configurations (Apache Real Web Servers Configuration)

On both web servers http1 and http2, apache should be running having a common serving file (for purpose of get checked by ldirectord).

yum install httpd -y
echo "webserverisworking" > /var/www/html/check.txt
service httpd start
chkconfig httpd on

Now, Create a loopback interface on each web server, so it doesn’t communicate with your network gateway/router directly.

vi /etc/sysconfig/network-scripts/ifcfg-lo:0

It must look like this:

DEVICE=lo:0
IPADDR=10.10.10.53
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback

vi /etc/sysctl.conf

It must look like this:

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2

sysctl -p
ifup lo:0

Windows XP/2003/2008 users!: if you are using IIS6/7 as a web server then you should follow the next steps otherwise just skip.

 

Cluster Nodes Configurations (IIS6/7 Real Web Server Configuration)

1. Create text file by using Notepad and name it "check.txt"
2. Fill this file with "webserverisworking" string.
3. Move file to "C:\inetpub\wwwroot" or anywhere your web files are.

If you are using Windows XP/2003 IIS web server then you should do these steps:

1. Install "Microsoft Loopback Adapter" by using "Add Hardware" icon in Control Panel.
2. Set IP to 10.10.10.53
3. Set Subnet Mask to 255.255.255.0
4. Don't Set Gateway or DNS
5. Done!

If you are using Windows 2008 IIS web servers then you should do these steps:

1. Install "Microsoft Loopback Adapter" by using "Add Hardware" icon in Control Panel.
2. Set IP to 10.10.10.53
3. Set Subnet Mask to 255.255.255.0
4. Don't Set Gateway or DNS
5. Then you need to use the following command line magic :

netsh interface ipv4 set interface "net" weakhostreceive=enabled
netsh interface ipv4 set interface "loopback" weakhostreceive=enabled
netsh interface ipv4 set interface "loopback" weakhostsend=enabled

Note: Obviously first you will need to rename the specific adapters from the default of "Local Area Network Connection 1" to either "net" or "loopback" respectively i.e.

See following link for more information http://blog.loadbalancer.org/direct-server-return-on-windows-2008-using-loopback-adpter/

 

Final Test

Use "ipvsadm" to list down current statistics of ldirectord. Make sure that both real servers IPs are listed there and have non-zero value in weight (since we’ve this default setup, it should be 1). If not, then try checking the log file, tcpdump on ldirector and apache logs on real servers. If everything works good, you’ll see changing content when browsing to http://10.10.10.53/ multiple times (from another system outside these cluster nodes). Then stop httpd on one web server, browse to the URL again and all requests should now be served from the other web server.

ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.10.10.53:http lblc
192.168.200.102:http             Route   1      0          0
192.168.200.103:http             Route   1      0          0

For more information use following commands:

ipvsadm -L -nc
ipvsadm -L -n --rate
ipvsadm -L -n --stats

 

References

http://bashukhan.com/ha-lb-cluster-on-centos5-without-actual-heartbeat-p
http://www.howtoforge.com/setting-up-a-high-availability-load-balancer-with-haproxy-heartbeat-on-debian-lenny
http://www.linuxvirtualserver.org/


Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Anonymous (not registered) on Wed, 2011-06-01 22:39.

Hello,

What happens if the load balance get down?

Submitted by please (registered user) on Sun, 2010-11-21 20:17.

Dear all

I need to setup 2 NIC on Load Balancer ?

How to route in Load Balancer for 10 network and 192 network?

 Load Balancer: ld.example.com, IP address: 10.10.10.52
Web Server 1: http1.example.com, IP address: 192.168.200.102
Web Server 2: http2.example.com, IP address: 192.168.200.103

Submitted by U Sa Yar Gyi (not registered) on Mon, 2012-03-12 09:01.
Good Question. Somebody Knows?
Submitted by sr71919 (not registered) on Wed, 2010-10-06 15:58.

how to add a public IP for failover node?

in  "/etc/ha.d/ldirectord.cf"

 logfile="/var/log/ldirectord.log"
quiescent=no
virtual=192.168.6.241:80
        fallback=127.0.0.1:80
        real=192.168.6.243:80 gate
        real=175.41.131.110:80 gate
  <snip>

 once if  192.168.6.243 (local node) fails, load balancer should redirect to the public IP 175.41.131.110.

 How do i do this public node failover?

Submitted by mbsouth (not registered) on Tue, 2010-02-16 08:34.

Thx for this tutorial, good job!

What I miss, in a lot of other tutorials too, is
- how to cross logging two or more Apache server when using it with LB
- how to synch the content of the Apache server when using it with LB
- how to repl. databases (not just mySQL, postgreSQL too) when using it with LB
 

mbsouth

Submitted by arfore (not registered) on Sun, 2010-08-22 01:20.

mbsouth:

 One solution for sychronization of Apache content would be to setup an NFS mount to a SAN or NAS to hold all the web content.  Then you would just mount that on each web node and use it as the webroot.  This is what I have setup at my current job.

 Of course, for it to keep with the HA aspect of this guide, you would need to setup up separate NFS nodes and lb them as well.  Not sure if NFS supports that.

 What I did was to write a simple script that looks for the existence of the NFS content and if doesn't exist then the system stops Apache, runs a script to drop an emergency config file in place of the normal one and restarts Apache so that an emergency webroot that is local to the node is used.  I have found that this works really well.

 arfore

Submitted by risker (not registered) on Thu, 2010-07-01 12:18.

Loadbalancing MySQL could be done in the same way with the help ldirectord

 -> Master/Master replication or

-> Mysql NDBCluster replication (attention: there are some traps/limitations of the ndbcluster engine!)

http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-limitations.html

Submitted by ellisgl (registered user) on Sun, 2011-07-03 15:34.
To do MySQL load balancing, I've used Galera from Codeship. It's a Master/Master setup that makes things realllllly simple. Load balancing is pretty easy.