Setting Up A High-Availability Load Balancer (With Failover and Session Support) With HAProxy/Heartbeat On Debian Etch

Want to support HowtoForge? Become a subscriber!
 
Submitted by falko (Contact Author) (Forums) on Tue, 2007-11-06 12:50. :: Debian | High-Availability

Setting Up A High-Availability Load Balancer (With Failover and Session Support) With HAProxy/Heartbeat On Debian Etch

Version 1.0
Author: Falko Timme <ft [at] falkotimme [dot] com>
Last edited 10/27/2007

This article explains how to set up a two-node load balancer in an active/passive configuration with HAProxy and heartbeat on Debian Etch. The load balancer sits between the user and two (or more) backend Apache web servers that hold the same content. Not only does the load balancer distribute the requests to the two backend Apache servers, it also checks the health of the backend servers. If one of them is down, all requests will automatically be redirected to the remaining backend server. In addition to that, the two load balancer nodes monitor each other using heartbeat, and if the master fails, the slave becomes the master, which means the users will not notice any disruption of the service. HAProxy is session-aware, which means you can use it with any web application that makes use of sessions (such as forums, shopping carts, etc.).

From the HAProxy web site: "HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net."

I do not issue any guarantee that this will work for you!

 

1 Preliminary Note

In this tutorial I will use the following hosts:

  • Load Balancer 1: lb1.example.com, IP address: 192.168.0.100
  • Load Balancer 2: lb2.example.com, IP address: 192.168.0.101
  • Web Server 1: http1.example.com, IP address: 192.168.0.102
  • Web Server 2: http2.example.com, IP address: 192.168.0.103
  • We also need a virtual IP address that floats between lb1 and lb2: 192.168.0.99

Here's a little diagram that shows our setup:

    shared IP=192.168.0.99
 192.168.0.100  192.168.0.101 192.168.0.102 192.168.0.103
 -------+------------+--------------+-----------+----------
        |            |              |           |
     +--+--+      +--+--+      +----+----+ +----+----+
     | lb1 |      | lb2 |      |  http1  | |  http2  |
     +-----+      +-----+      +---------+ +---------+
     haproxy      haproxy      2 web servers (Apache)
     heartbeat    heartbeat

The shared (virtual) IP address is no problem as long as you're in your own LAN where you can assign IP addresses as you like. However, if you want to use this setup with public IP addresses, you need to find a hoster where you can rent two servers (the load balancer nodes) in the same subnet; you can then use a free IP address in this subnet for the virtual IP address. Here in Germany, Hetzner is a hoster that allows you to do this - just talk to them. Update: Hetzner's policies have changed - please read here for more details: http://www.howtoforge.com/forums/showthread.php?t=19988

http1 and http2 are standard Debian Etch Apache setups with the document root /var/www (the configuration of this default vhost is stored in /etc/apache2/sites-available/default). If your document root differs, you might have to adjust this guide a bit.

To demonstrate the session-awareness of HAProxy, I'm assuming that the web application that is installed on http1 and http2 uses the session id JSESSIONID.

 

2 Preparing The Backend Web Servers

We will configure HAProxy as a transparent proxy, i.e., it will pass on the original user's IP address in a field called X-Forwarded-For to the backend web servers. Of course, the backend web servers should log the original user's IP address in their access logs instead of the IP addresses of our load balancers. Therefore we must modify the LogFormat line in /etc/apache2/apache2.conf and replace %h with %{X-Forwarded-For}i:

http1/http2:

vi /etc/apache2/apache2.conf

[...]
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
[...]

Also, we will configure HAProxy to check the backend servers' health by continuously requesting the file check.txt (translates to /var/www/check.txt if /var/www is your document root) from the backend servers. Of course, these requests would totally bloat the access logs and mess up your page view statistics (if you use a tool like Webalizer or AWstats that generates statistics based on the access logs).

Therefore we open our vhost configuration (in this example it's in /etc/apache2/sites-available/default) and put these two lines into it (comment out all other CustomLog directives in your vhost configuration):

vi /etc/apache2/sites-available/default

[...]
SetEnvIf Request_URI "^/check\.txt$" dontlog
CustomLog /var/log/apache2/access.log combined env=!dontlog
[...]

This configuration prevents that requests to check.txt get logged in Apache's access log.

Afterwards we restart Apache:

/etc/init.d/apache2 restart

... and create the file check.txt (this can be an empty file):

touch /var/www/check.txt

We are finished already with the backend servers; the rest of the configuration happens on the two load balancer nodes.

 

3 Installing HAProxy

Unfortunately HAProxy is available as a Debian package for Debian Lenny (testing) and Sid (unstable), but not for Etch. Therefore we will install the HAProxy package from Lenny. To do this, open /etc/apt/sources.list and add the line deb http://ftp2.de.debian.org/debian/ lenny main; your /etc/apt/sources.list could then look like this:

lb1/lb2:

vi /etc/apt/sources.list

deb http://ftp2.de.debian.org/debian/ etch main
deb-src http://ftp2.de.debian.org/debian/ etch main

deb http://ftp2.de.debian.org/debian/ lenny main

deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib

Of course (in order not to mess up our system), we want to install packages from Lenny only if there's no appropriate package from Etch - if there are packages from Etch and Lenny, we want to install the one from Etch. To do this, we give packages from Etch a higher priority in /etc/apt/preferences:

vi /etc/apt/preferences

Package: *
Pin: release a=etch
Pin-Priority: 700

Package: *
Pin: release a=lenny
Pin-Priority: 650

(The terms etch and lenny refer to the appropriate terms in /etc/apt/sources.list; if you're using stable and testing there, you must use stable and testing instead of etch and lenny in /etc/apt/preferences as well.)

Afterwards, we update our packages database:

apt-get update

... upgrade the installed packages:

apt-get upgrade

... and install HAProxy:

apt-get install haproxy

 

4 Configuring The Load Balancers

The HAProxy configuration is stored in /etc/haproxy.cfg and is pretty straight-forward. I won't explain all the directives here; to learn more about all options, please read http://haproxy.1wt.eu/download/1.3/doc/haproxy-en.txt and http://haproxy.1wt.eu/download/1.2/doc/architecture.txt.

We back up the original /etc/haproxy.cfg and create a new one like this:

lb1/lb2:

cp /etc/haproxy.cfg /etc/haproxy.cfg_orig
cat /dev/null > /etc/haproxy.cfg
vi /etc/haproxy.cfg

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        #debug
        #quiet
        user haproxy
        group haproxy

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

listen webfarm 192.168.0.99:80
       mode http
       stats enable
       stats auth someuser:somepassword
       balance roundrobin
       cookie JSESSIONID prefix
       option httpclose
       option forwardfor
       option httpchk HEAD /check.txt HTTP/1.0
       server webA 192.168.0.102:80 cookie A check
       server webB 192.168.0.103:80 cookie B check

Afterwards, we set ENABLED to 1 in /etc/default/haproxy:

vi /etc/default/haproxy

# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1
# Add extra flags here.
#EXTRAOPTS="-de -m 16"

Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by matthias (not registered) on Thu, 2012-10-04 13:23.

Falko ich bin ein riesen Fan von dir. Nur in dem tut hätte mir gefehlt, dass cookie JSESSIONID prefix geändert werden muss in die entsprechende Session die von php/java benutzt wird. 


 Gruß 

 

Submitted by toodles (registered user) on Mon, 2007-12-10 20:31.

Hi,

I just ordered two servers from Hetzner and asked for an extra nic and crossover cable in each. Having figured that I could setup a HA server, I sent a request about this. They gave me a very clear response message that they no longer offer that. Their ip routing now uses mac address filtering. It might be a good idea to remove that from the tutorial before somebody else wastes money on servers. Hetzner servers are very cool, but cannot be used for HA anymore.

Regards,

Tim 

Submitted by admin (registered user) on Tue, 2007-12-11 09:32.
I'd call them and ask for the telephone number of the data center guys - these guys can help you; everyone else will tell you that virtual IP addresses are not possible.
Submitted by toodles (registered user) on Fri, 2007-12-14 15:48.

Just want to give an update on this matter. If the servers are placed in the new data centre (Number 8), virtual or shared ip address are no longer possible. This is only possible in their other data centres.

Just letting you know so that others don't make the mistake I made, and just assume it will work. (I really must learn to make less assumtions!)

Having said that, they are trying to find room in one of the other data centres for me now. I really really hope they do :-) and I just wanted to say thank you for suggesting to talk to the datacentre guys directly. They are certainly A LOT more helpful!! 

Submitted by toodles (registered user) on Fri, 2007-12-14 04:14.

Thanks, I'll try that. I heard they introduced mac filtering and for that reason this was no longer possible. When I called them, I was told I need to talk to the root server techies and was given a number. I assumed that these were datacentre guys, but it seems that they're not. I'll let you know how I get on after I talk to the *real* people ;-)