Console Login

Active-passive HAProxy and keepalived load balancing (2009)

The problem

A small web site on one virtual server stops when that server stops. A kernel panic, a full disk, a failed package upgrade or a host-node reboot all take the site offline until someone logs in. Scaling up the single machine does not help, because the machine is still the only one.

The usual first answer is to put a load balancer in front of several web servers. That moves the problem rather than removing it. A common 2009 layout has one HAProxy node and two to four Apache nodes behind it. If the HAProxy node fails, every web node is unreachable, however healthy it is. Content is often kept in sync with rsync or NFS, and an NFS server shared by all nodes is a second single point of failure.

The target of "100% uptime" is not achievable. A realistic target is to keep a failure of any one machine from becoming an outage, and to make recovery take seconds instead of hours.

Why it happens

The cause is the topology, not any one piece of software. Every request passes through a component that exists exactly once:

  • One entry point. The public IP address is bound to one machine. Clients and DNS records point at that address, so nothing else can answer for it.
  • One shared data service. NFS or a single database server is used by every web node. Adding web nodes adds capacity but not availability for that service.
  • No health checking. Without checks, a balancer keeps sending a share of traffic to a dead backend. Visitors see errors even though other nodes are fine.

The virtualisation type also matters. Container-based virtual servers such as OpenVZ share the host kernel. A guest in such a container normally cannot add addresses to an interface or send the gratuitous ARP that an address takeover needs. Full virtualisation (Xen, KVM, VMware) gives each guest its own kernel and network stack, so these tools work as on a physical server. The two are often confused when people compare "VPS" and "VDS". For this design, what counts is whether the guest controls its own network interface.

Source of truth

Two references define the behaviour this fix relies on.

  • RFC 3768, Virtual Router Redundancy Protocol (VRRP) version 2. It confirms that several routers can share one virtual router ID and virtual IP address, and that the node with the highest priority is elected master and owns the address. The master sends periodic advertisements, by default once per second. If the backups stop receiving them for about three advertisement intervals, they elect a new master. In practice, failover takes a few seconds. The advertisements are IP protocol 112 sent to multicast group 224.0.0.18. The RFC also defines the simple password authentication used in the configuration below.
  • HAProxy 1.3 configuration manual. It documents the listen section, balance roundrobin, option httpchk for HTTP health checks, the check, inter, rise and fall server parameters, option redispatch, and option forwardfor. The last one adds the client address in an X-Forwarded-For header.

keepalived is the Linux implementation of VRRP used here. Its project site is keepalived.org.

Ways to fix it

1. Active-passive pair sharing a floating IP (VRRP)

Two identical nodes run the same service. One holds the public address, and the other takes it over when the first stops answering. keepalived implements VRRP. Heartbeat from the Linux-HA project reaches the same goal with its own cluster messaging and resource scripts, and it also supports shared-storage resources.

2. Load balancer pair plus several web nodes

The floating-IP pair runs HAProxy. HAProxy distributes requests to two or more web nodes and removes failed nodes from rotation. This is the layout used below. It protects both the entry point and the web tier.

3. DNS round robin

Publish several A records, one per web node. It needs no extra software. It has no health checking, and resolver caching keeps clients on a dead address long after the record is changed, so recovery is slow and uneven.

4. Provider-side IP failover

Some hosting platforms can move an address between guests through an API or control panel. This is the only workable option on container guests that have no control over their own interfaces. Failover is usually triggered by a script or by hand, not by the guests.

5. Hardware load balancers

Commercial appliances combine all of this in one product. They cost more, and they are themselves deployed in pairs.

Why this fix

Option 2 is chosen: two HAProxy nodes with keepalived, in front of at least two web nodes. The reasons:

  • It removes both the entry-point and the web-tier single points of failure with software that is free and widely documented.
  • Health checks take failed web nodes out of rotation automatically, which DNS round robin cannot do.
  • VRRP failover takes seconds, as the RFC describes, and clients keep using the same address.
  • Capacity grows by adding a server line to the HAProxy configuration, with no change visible to clients.

The design needs guests that control their own network stack: full virtualisation, or dedicated hardware. On container guests, use option 4 instead.

Shared data still needs its own plan. Keep the web nodes stateless where possible. Push the content to each node with rsync from a deploy host, instead of having all nodes read one NFS export. Run the database on a separate node and allow connections to it from the web nodes only.

Step by step

The steps below assume four CentOS 5 era guests: lb1 and lb2 as balancers, web1 (10.0.0.11) and web2 (10.0.0.12) as Apache nodes. The shared public address is 192.0.2.10 (a documentation address, to be replaced with a real one). These commands and files are given as instructions. No output from running them is reproduced, and no measurements are claimed.

1. Web nodes: a health page and the real client address

Create a small static file that the balancer can request, and push content from the deploy host.

echo ok > /var/www/html/health.html
rsync -az --delete -e ssh /var/www/site/ deploy@web1:/var/www/html/
rsync -az --delete -e ssh /var/www/site/ deploy@web2:/var/www/html/

Behind a proxy, Apache sees the balancer's address as the client. To log the real address, replace %h with %{X-Forwarded-For}i in the LogFormat line, then reload.

service httpd reload

2. Balancer nodes: install and allow non-local binds

The standby node does not own the virtual address, so HAProxy must be allowed to bind to it anyway. Install keepalived from the distribution, and build HAProxy from the 1.3 source tree if no package is available.

yum install keepalived
groupadd haproxy
useradd -g haproxy -s /sbin/nologin haproxy
make TARGET=linux26
make install
echo 'net.ipv4.ip_nonlocal_bind = 1' >> /etc/sysctl.conf
sysctl -p

3. Balancer nodes: HAProxy configuration

Use the same file on both nodes, at /etc/haproxy/haproxy.cfg.

global
    log 127.0.0.1 local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option forwardfor
    option redispatch
    retries 3
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000

listen www 192.0.2.10:80
    balance roundrobin
    option httpchk GET /health.html
    server web1 10.0.0.11:80 check inter 2000 rise 2 fall 3
    server web2 10.0.0.12:80 check inter 2000 rise 2 fall 3

Check the file before starting the service.

haproxy -c -f /etc/haproxy/haproxy.cfg
haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid

4. Balancer nodes: keepalived configuration

On lb1, write /etc/keepalived/keepalived.conf as follows. On lb2, use the same file with priority 100. The check script adds 2 to the priority while HAProxy is running. If HAProxy dies on the master, its effective priority drops below the backup's and the address moves, even though the machine itself is alive.

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass replace-me
    }
    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }
    track_script {
        chk_haproxy
    }
}

Allow VRRP through the firewall on both balancers, then start the service.

iptables -I INPUT -p 112 -d 224.0.0.18 -j ACCEPT
service iptables save
chkconfig keepalived on
service keepalived start

5. Test the failover

Confirm which node holds the address, request the site through it, then stop keepalived on the master and repeat on the other node.

ip addr show eth0
curl -I http://192.0.2.10/
service keepalived stop

Also stop Apache on web1 and request the site repeatedly. After the health check marks it down (about three failed checks, six seconds with the settings above), all requests should be answered by web2.

service httpd stop
curl -I http://192.0.2.10/

Review

What the design gives. Loss of any one balancer or web node no longer takes the site down. Failover needs a few seconds, set by the VRRP advertisement interval and the health check settings. Both values can be tuned, with the trade-off that very short intervals cause false failovers on a busy or briefly stalled guest.

Limits.

  • Only one balancer is active. The second is idle capacity, and a single HAProxy node is the throughput ceiling.
  • Sessions kept in memory on one web node are lost when it fails. Keep sessions in a shared store, or use cookie-based persistence in HAProxy and accept the loss on failure.
  • The database is still a single point of failure unless it is replicated separately. For MySQL in this era that means master-slave replication, often with a manual promotion step.
  • VRRP needs layer-2 adjacency between the balancers and guests that control their interfaces. Container guests and networks that block multicast need provider-side IP failover instead.
  • Using password authentication in VRRP only prevents accidents. It does not protect against an attacker on the same network segment.

Other choices. Heartbeat fits when resources beyond an IP address, such as a replicated disk or a database, must fail over together. Hardware balancers fit when support contracts are required. The choice of a hosting country or provider decides which data protection rules apply, but it has no effect on availability. Treat it as a separate decision. For administration, expose SSH only to trusted addresses or through a VPN, and keep the database reachable only from the web nodes.