The problem
A single server that runs the web server, the application runtime and the database fails as one unit. A slow query that holds a MyISAM table lock, or an Apache prefork pool that consumes all CPU during a traffic peak, stalls every request, however much RAM the machine has. Adding a second web node does not remove the risk by itself, and typical symptoms appear:
- All pages slow down together when one expensive request type piles up.
- Patching or restarting the web service means visible downtime.
- With plain DNS round robin, clients keep being sent to a dead node until the record expires.
- Once a load balancer is added, that one machine becomes the next single point of failure.
- Under heavy load the balancer itself starts to refuse or reset connections although CPU is idle.
Why it happens
Several independent causes overlap:
- Shared resources. Web, application and database processes compete for the same CPU, memory and disk. MyISAM uses table-level locks, so a long write or a long read can block all other statements on that table.
- No health awareness in front of the nodes. DNS round robin hands out addresses without knowing whether the target answers.
- One balancer. A proxy with a single address fails completely if the host or the process dies.
- Kernel defaults. A proxy opens one outgoing connection to a backend for every client connection. On CentOS 6 the default ephemeral port range is roughly 32768 to 61000, so only about 28,000 source ports are available per backend address and port, and closed connections stay in TIME_WAIT and keep occupying ports. A busy balancer with two backends can run out of local ports long before it runs out of CPU.
Source of truth
The behaviour of the load balancer is defined by the HAProxy 1.4 configuration manual: HAProxy 1.4 Configuration Manual. It documents the balance algorithms (roundrobin, leastconn, source and others), option httpchk for HTTP health checks, the check, inter, rise and fall server keywords, option forwardfor, option http-server-close and the stats directives including stats auth. It also states that leastconn is intended for long-lived sessions (LDAP, SQL and similar) and is less suited to short protocols such as HTTP, which matters for the algorithm choice below.
The failover behaviour of the virtual address follows VRRP, specified in RFC 3768 (VRRP version 2). It defines the master and backup roles, priorities from 1 to 254, the advertisement interval, and the rule that a backup takes over when it has not heard an advertisement for three advertisement intervals plus a skew time that depends on its priority.
The kernel parameters are described in the Linux ip-sysctl documentation. It confirms that tcp_tw_reuse allows reuse of TIME_WAIT sockets for new outgoing connections, which is the case that matters for a proxy, and that ip_nonlocal_bind lets a process bind to an address the host does not currently own.
Ways to fix it
Distributing the traffic
| Method | Strengths | Weaknesses |
|---|---|---|
| DNS round robin | No extra component | No health checks, caching delays changes |
| Nginx upstream | Already present on many web nodes, can terminate SSL | Only passive failure detection in the open-source build, no built-in per-server status page |
| HAProxy 1.4 | Active health checks, queueing, per-server statistics, TCP and HTTP modes | No SSL termination in 1.4, one more service to run |
| LVS / IPVS | Kernel-level layer 4 balancing, very low overhead | No HTTP awareness, more complex setup |
| Hardware appliance | Vendor support | Cost, less flexible |
Removing the balancer as a single point of failure
- Keepalived with VRRP: two balancers share one virtual IP; the backup claims it when advertisements stop.
- Heartbeat or Pacemaker: a full cluster stack with resource management; more capable and considerably heavier.
- DNS failover: low-TTL records changed by a monitor; failover time is bounded by client and resolver caching.
The chosen fix
The choice is HAProxy 1.4 for balancing, with two nodes using Keepalived for the shared address. HAProxy gives active health checks and a statistics page without extra modules, and Keepalived is a small daemon that packages cleanly on CentOS 6 and does one job. Heartbeat or Pacemaker is justified only when other resources must fail over together.
For the algorithm, leastconn suits backends where request duration varies a lot, because new requests go to the server with the fewest connections in progress. The manual's caveat applies, though: roundrobin is not wrong for HTTP. With uniform, short requests it behaves just as well and is more predictable. A reasonable default is leastconn with option http-server-close, and a switch to roundrobin if load becomes uneven.
Two points of the common tutorial setup need correction. The stats auth keyword needs a space before the credentials, and the statistics page should not sit on the public listener. Also, an ACL that is defined but never referenced by a use_backend rule has no effect and is left out below.
Step by step
The commands below are instructions for a CentOS 6.3 system with two balancer nodes (the first has the address 10.0.0.1, the second 10.0.0.4) and two web nodes on the private network (10.0.0.2 and 10.0.0.3). Adjust addresses and the interface name. No command output is reproduced here.
1. Install the packages
On CentOS 6.3 HAProxy 1.4 comes from the EPEL repository, so enable that repository first. Keepalived is in the base repository.
yum install haproxy keepalived
chkconfig haproxy on
chkconfig keepalived on
2. Tune the kernel
Append to /etc/sysctl.conf. The range avoids the highest ports, tcp_tw_reuse helps the outgoing backend connections, and ip_nonlocal_bind lets HAProxy start on the node that does not currently hold the virtual address. Do not enable tcp_tw_recycle: it breaks clients behind NAT.
fs.file-max = 100000
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_nonlocal_bind = 1
sysctl -p
3. Enable HAProxy logging
HAProxy sends logs over UDP to syslog. In /etc/rsyslog.conf enable the UDP input on the loopback address only, and add a rule file.
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
# /etc/rsyslog.d/haproxy.conf
local2.* /var/log/haproxy.log
service rsyslog restart
4. Configure HAProxy
Use the same /etc/haproxy/haproxy.cfg on both balancers. Each proxied connection uses two sockets, and HAProxy derives its file descriptor limit from maxconn by itself.
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main_http_in
bind *:80
default_backend app_servers
backend app_servers
balance leastconn
option httpchk GET /
server web01 10.0.0.2:80 check inter 2000 rise 2 fall 3
server web02 10.0.0.3:80 check inter 2000 rise 2 fall 3
listen stats 10.0.0.1:8404
stats enable
stats uri /haproxy?stats
stats auth admin:CHANGE_THIS_PASSWORD
With inter 2000 and fall 3 a dead backend is removed after about six seconds. The httpchk target should be a cheap URL that exists on the web nodes; a dedicated health page is better than /. On the second balancer the statistics listener must use its own address. The stats credentials travel unencrypted, so restrict the port to the private network.
Validate the file and start the service:
haproxy -c -f /etc/haproxy/haproxy.cfg
service haproxy start
5. Configure Keepalived
The health script lowers the priority when the HAProxy process disappears, so the address moves even though the host is still up. With the weights below, a healthy master has priority 103, a master without HAProxy has 101, and a healthy backup has 102, which causes the switch. Write the following to /etc/keepalived/keepalived.conf on the master:
vrrp_script chk_haproxy {
script "/sbin/pidof haproxy"
interval 2
weight 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass s3cr3t01
}
virtual_ipaddress {
192.168.10.50/24 dev eth0
}
track_script {
chk_haproxy
}
}
On the backup node use the same file with state BACKUP and priority 100. The auth_pass value is sent in clear text and is limited to eight characters, so it only protects against misconfiguration, not against attackers on the segment.
6. Allow VRRP and start the service
VRRP uses IP protocol 112 and the multicast address 224.0.0.18. If iptables is active, allow it on both nodes.
iptables -I INPUT -p 112 -d 224.0.0.18 -j ACCEPT
service iptables save
service keepalived start
7. Test failover
Check which node holds the address, request a page through it, stop HAProxy on the master and check again on the backup. The address should appear on the backup after a few seconds, and requests to 192.168.10.50 should keep working.
ip addr show eth0
curl -I http://192.168.10.50/
service haproxy stop
ip addr show eth0
Afterwards restart HAProxy on the master with service haproxy start. The master priority rises again and it takes the address back.
Review
The configuration gives active health checks, removal of a failed web node within seconds, the ability to take a node out for maintenance by stopping its web service, and a balancer pair without a single point of failure. The kernel changes remove the most common cause of connection failures on a busy proxy, port exhaustion.
Limits to keep in mind:
- HAProxy 1.4 does not terminate SSL. Pass HTTPS through in
mode tcp, or terminate it in front with Nginx or stunnel, which loses the original client address unless it is forwarded separately. - Applications that keep sessions in local files or memory need sticky sessions (a
cookieline on the backend and a cookie value on eachserverline) or a shared session store. - Keepalived relies on multicast VRRP. Some virtualization platforms and switches block it, and then both nodes may claim the address. Test this on the actual network before relying on it.
- The database is still a single component. Balancing the web tier does not remove it; replication and a separate failover plan are needed.
- For simple setups with one or two backends and SSL on the same host, Nginx upstream may be enough. For pure layer 4 throughput, LVS is an alternative, and for clusters with many dependent resources, Pacemaker is the better fit.