The problem
Splitting an application into services replaces in-process calls with network calls. A typical setup then hardcodes backend addresses in /etc/hosts, in an application config file, or in a static Nginx or HAProxy backend block. That works for two web servers. It stops working when instances are added, removed or replaced often, because:
- A new instance receives no traffic until someone edits a file on every client host.
- A dead or hung instance keeps receiving requests until someone removes it by hand.
- Different hosts end up with different ideas of which instances exist.
The routing layer therefore needs three things: registration (an instance announces itself), discovery (clients learn the current list) and health checking (traffic stops flowing to a node that hangs).
Why it happens
Static configuration describes a topology that was true at one moment. Nothing in a hardcoded file reacts when a process dies, a host reboots or an instance is added. Two further causes make a naive fix fail in practice.
- Registry entries that outlive the service. If an instance writes its address into a registry and nothing removes it after a crash, the registry becomes another stale static file.
- Misread HAProxy settings. Some commonly copied settings do not do what their authors assume.
timeout queueis not a circuit breaker. It only limits how long a request waits for a free connection slot, and the queue only forms when a server has amaxconnlimit.option redispatchlets a request leave a server it was bound to, for example by a cookie, and be retried on another server after connection failures. It does not remove a failing server from rotation. That job belongs tocheck.
The registry itself is also a hazard. ZooKeeper acknowledges writes only after they reach its transaction log on disk, and sessions expire on timers. Slow disk writes, swapping or a starved CPU on the ensemble hosts can therefore cause session timeouts and leader re-elections. Run the ensemble on hosts with predictable disk latency and dedicated CPU and memory. A full virtual machine with its own kernel is easier to reason about than a heavily shared container host.
Source of truth
The fix relies on three documented behaviours:
- ZooKeeper Programmer's Guide (3.4.6) describes ephemeral znodes, which are deleted automatically when the creating client's session ends. It also describes watches, which are one-time triggers that must be re-registered after they fire. Ephemeral znodes cannot have children, so each instance entry has to be a leaf.
- HAProxy 1.4 Configuration Manual defines
checkwithinter,riseandfall,option httpchk,retries,option redispatch,timeout queue,maxconnand the-sfsoft-reload behaviour. In that mode a new process takes over the listening sockets and asks the old process to finish its existing connections and exit. - Synapse and Nerve are the open-source tools behind the SmartStack pattern. Nerve registers healthy services in ZooKeeper, and Synapse reads ZooKeeper and rewrites the local HAProxy configuration. Both repositories document the pattern this article reproduces by hand.
- Kazoo watcher recipes document
ChildrenWatch, which re-registers the ZooKeeper watch automatically and calls a function with the current child list on every change.
Fix options
1. Configuration management
Puppet or Chef renders the HAProxy backend from an inventory and runs periodically. It is simple and reviewable, but changes only appear after the next run. The inventory also has no idea whether a process is alive.
2. A local watcher on a ZooKeeper tree
Each instance creates an ephemeral znode. A small daemon on every host watches the tree, renders the backend section, validates the file and reloads HAProxy gracefully. HAProxy's own health checks still decide, within seconds, whether a registered instance receives traffic.
3. Synapse and Nerve
The same design as option 2, as maintained tools. They include more discovery methods, templating and registration with health checks, at the price of a Ruby runtime and more moving parts.
4. HAProxy's runtime socket only
The stats socket can disable or enable existing servers and change weights. It cannot add a new server line, so it complements a reload-based approach and does not replace one.
Why this fix
Option 2 is the chosen method. It keeps the data path simple: applications talk to 127.0.0.1, and the local HAProxy does the routing. The control path is small enough to read in one sitting. Two properties matter most:
- Ephemeral registration means a crashed instance disappears from the registry without a cleanup job.
- Local rendering means each host keeps its last good configuration if ZooKeeper becomes unreachable. HAProxy continues to route and health-check on its own, so a registry outage freezes the topology instead of breaking traffic.
Synapse and Nerve are the better choice once several services, templates or non-HTTP backends are involved. The hand-written watcher is a good way to understand what they do.
Applying the fix
The steps below are instructions only. Nothing here was executed for this article, so no program output is reproduced. Compare real output on a test host with the expectations stated in the text.
ZooKeeper ensemble
Use an odd number of servers. Three tolerate one failure, because a majority must stay up. Put the transaction log on its own device if possible. Each host needs a myid file whose number matches its server.N line.
# /etc/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper-log
clientPort=2181
server.1=10.0.0.2:2888:3888
server.2=10.0.0.3:2888:3888
server.3=10.0.0.4:2888:3888
Each server should answer a health probe and report its role. A healthy server replies imok to the first command, and Mode: in the second shows leader or follower.
echo ruok | nc 10.0.0.2 2181
echo stat | nc 10.0.0.2 2181
Registration with an ephemeral znode
The znode must be created by a process that stays connected for as long as the service is healthy. It should be created only after the service's own /health endpoint succeeds. Use the same address space as the HAProxy servers, here 10.0.0.x.
import time
from kazoo.client import KazooClient
zk = KazooClient(hosts='10.0.0.2:2181,10.0.0.3:2181,10.0.0.4:2181')
zk.start()
zk.create('/services/inventory_service/instances/10.0.0.5:8080',
b'', ephemeral=True, makepath=True)
while True:
time.sleep(1)
The registered tree then looks like this. Inspect it with zkCli.sh.
/services
/inventory_service
/instances
/10.0.0.5:8080
/10.0.0.6:8080
zkCli.sh -server 10.0.0.2:2181 ls /services/inventory_service/instances
Static part of the HAProxy configuration
Save this as /etc/haproxy/haproxy.cfg.head. The watcher appends the generated backend to it. A dead host that silently drops packets can cost up to timeout connect per attempt, so keep that value low for internal calls. A server is marked down after three failed checks, about 6 seconds with a 2 second interval plus one check timeout. It needs two successful checks, about 4 seconds, to return.
global
daemon
maxconn 4096
log 127.0.0.1 local0
pidfile /var/run/haproxy.pid
defaults
mode http
log global
retries 3
option redispatch
timeout connect 1000ms
timeout check 1000ms
timeout client 30000ms
timeout server 30000ms
frontend local_inventory
bind 127.0.0.1:8081
default_backend inventory_backend
Watcher that renders, validates and reloads
The original idea of calling an undefined generator is replaced by a complete script. It writes to a temporary file, lets HAProxy validate it with -c and only then replaces the live file and performs the soft reload.
import os
import subprocess
import time
from kazoo.client import KazooClient
ZK_HOSTS = '10.0.0.2:2181,10.0.0.3:2181,10.0.0.4:2181'
PATH = '/services/inventory_service/instances'
CFG = '/etc/haproxy/haproxy.cfg'
HEAD = '/etc/haproxy/haproxy.cfg.head'
PID = '/var/run/haproxy.pid'
def render(children):
lines = ['backend inventory_backend',
' balance roundrobin',
' option httpchk GET /health',
' timeout queue 1000ms']
for i, addr in enumerate(sorted(children)):
lines.append(' server inventory_%02d %s check inter 2000 rise 2 fall 3 maxconn 100'
% (i + 1, addr))
return '\n'.join(lines) + '\n'
def apply_config(children):
tmp = CFG + '.new'
with open(tmp, 'w') as f:
f.write(open(HEAD).read() + render(children))
if subprocess.call(['haproxy', '-c', '-f', tmp]) != 0:
return
os.rename(tmp, CFG)
subprocess.call('haproxy -f %s -p %s -sf $(cat %s)' % (CFG, PID, PID),
shell=True)
zk = KazooClient(hosts=ZK_HOSTS)
zk.start()
zk.ensure_path(PATH)
@zk.ChildrenWatch(PATH)
def watch_instances(children):
apply_config(children)
while True:
time.sleep(1)
Run the script under a supervisor so it restarts after a crash. If the first reload happens before any HAProxy process exists, cat of a missing pid file fails. Start HAProxy once manually with the same -f and -p options.
haproxy -c -f /etc/haproxy/haproxy.cfg
haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
curl -i http://127.0.0.1:8081/
Limits and alternatives
The design gives automatic registration, health-based removal and a graceful reload. Expect these limits:
- Reloads have a cost. Every change starts a new HAProxy process. Under constant churn, batch changes (debounce the watcher) instead of reloading per event. Established connections finish on the old process, so long-lived connections can keep it alive for a while.
- A live process is not a healthy service. An ephemeral znode only proves the registering process is connected to ZooKeeper. The HAProxy
httpchkis what proves the service answers. Keep both. - Retries are not free.
retriesandredispatchapply to connection failures. Do not assume HAProxy will replay a request that was accepted and then failed, especially a non-idempotent one. - Stale-but-working is deliberate. If ZooKeeper is down, hosts keep the last configuration. New instances stay invisible until it returns.
- Traffic between hosts is not encrypted. HAProxy 1.4 does not terminate TLS. If service traffic crosses untrusted links, wrap it with stunnel or a VPN. Personal data moving between services also falls under the applicable data protection rules, so keep hosts and the ZooKeeper ensemble in the jurisdiction those rules require.
For a handful of stable backends, configuration management is enough. For many services with frequent changes, prefer Synapse and Nerve over maintaining this script. Both fit the 2014 toolset: HAProxy 1.4 or 1.5 development releases, ZooKeeper 3.4 and kazoo.