Console Login

Raft control planes: fsync latency for etcd, Swarm and Nomad

The problem

A small self-managed cluster runs fine during a quiet test, then starts to misbehave under ordinary load. A typical set of symptoms:

  • The Kubernetes API server answers slowly or returns timeouts, and kubectl commands hang.
  • The control plane changes leader repeatedly, so deployments stall or are rejected for a few seconds at a time.
  • The etcd log contains warnings such as failed to send out heartbeat on time, apply entries took too long or wal: sync duration of ... expected less than 1s.
  • Liveness probes fail on healthy containers, and the orchestrator restarts them, which adds load and produces a cascade.

These warnings are written by etcd itself, so look for them in the etcd container or service log. They do not appear in the kube-apiserver log. The same pattern shows up on Docker Swarm (managers lose quorum or change leader) and on Nomad (servers re-elect), because all three keep cluster state in a Raft-replicated store.

Why it happens

The orchestrators differ a lot in features, but their control planes share one design: a small group of members agree on every state change through the Raft consensus protocol. A write is accepted only after the leader has flushed it to its own disk, sent it to the followers, and a majority has flushed it too. The flush is an fsync or fdatasync call, so write latency to stable storage becomes part of every cluster operation.

OrchestratorWhere cluster state livesConsequence
Kubernetesetcd, usually on three control plane nodesSlow fsync delays API writes and leader liveness
Docker SwarmRaft log on the manager nodes, built into the engineSlow managers lose leadership; use an odd number of managers
NomadRaft on the server agentsSame sensitivity, with a smaller footprint

Two infrastructure conditions turn this design into an outage:

  • Slow or contended storage. Spinning disks, throttled volumes, network-attached disks, and local disks shared with a busy database or log writer all show high fsync tail latency.
  • Stolen CPU. On an oversold hypervisor a guest is paused while it waits for a physical core. Heartbeats and health probes then miss their deadlines even though the disk is fine. Top and vmstat report this as the st value.

A third factor is the time budget. Heartbeat and election timeouts are fixed values, so a leader whose disk stalls for longer than the election timeout is effectively down. Peers elect a new leader even though the old one still answers on the network.

Source of truth

The etcd project documents this behaviour in its FAQ, and the guidance applies to every Raft-based control plane:

  • etcd FAQ (v3.4) confirms that leader loss from disk latency spikes is intentional, because disk latency is part of leader liveness. It lists a slow or shared virtualized disk as the usual cause of the heartbeat and slow-apply warnings. It names fio as the tool to judge whether a disk is fast enough. It gives the p99 targets: backend_commit_duration_seconds under 25 ms.
  • etcd FAQ (v3.3) recommends SSDs and gives a p99 target of under 10 ms for wal_fsync_duration_seconds. It says the usual cure for a slow disk is a dedicated or faster disk.
  • etcdHighFsyncDurations runbook explains why every write needs several fsyncs under Raft. It notes that etcd performs poorly on network-attached storage and uses the same 10 ms p99 threshold for alerting.
  • etcd performance guidance (OKD) shows a fio-based check that reports whether the 99th percentile of fsync is within a 20 ms limit. That is a looser limit than the 10 ms in the etcd FAQ, so treat 10 ms as the target and 20 ms as a hard ceiling.

Fix options

  1. Move the control plane data to fast local storage. Use local SSD or NVMe rather than network volumes or spinning disks. Keep the etcd data directory (or the Swarm and Nomad data directories) off the disk used by noisy workloads.
  2. Dedicate a disk or a node. Give the control plane its own device, or run control plane nodes without general workloads, so application I/O cannot compete with Raft writes.
  3. Raise the I/O priority of etcd. Put the etcd process into a higher I/O scheduling class with ionice. This helps on shared disks, but it only reorders requests.
  4. Relax the timing parameters. Increase etcd --heartbeat-interval and --election-timeout. This tolerates slow disks and links at the cost of slower failover. The election timeout should stay several times larger than the heartbeat interval.
  5. Reduce write load. Fewer objects, fewer frequent updates, and, on very large Kubernetes clusters, a separate etcd cluster for events.
  6. Pick a lighter orchestrator. Swarm or Nomad needs fewer moving parts than Kubernetes. They still use Raft, so this reduces overhead but does not remove the disk requirement.

Why this fix

Start with measurement, then fix the storage. Option 1 removes the cause. Options 3 and 4 only hide it: higher timeouts make a leader that is slow by a fixed margin look healthy, and they delay real failure detection. They are worth using as a secondary safety margin, not as the primary remedy. Option 6 is a design decision that should follow requirements, not disk problems. Kubernetes gives the richest ecosystem (operators, granular RBAC, stateful workloads), Swarm is the shortest path from a Compose file to a multi-node service, and Nomad schedules non-container workloads as well as containers.

The measurement step also separates the two infrastructure causes. A disk that fails the fsync test needs different action from a guest that shows steal time, and both can be tested in a few minutes before any orchestrator is installed.

Step by step

The commands below are shown as a procedure to apply. Output is not reproduced here because it depends entirely on the host. Compare your own numbers against the thresholds in the source section.

1. Test fsync latency on the disk that will hold the state

Use a scratch directory on the same filesystem as the etcd data directory. The 22 MiB size and 2300-byte block size are a widely used approximation of etcd write-ahead-log traffic, not an official specification. The test calls fdatasync after each write.

mkdir -p /var/lib/etcd-fio-test
fio --name=etcd-fsync --directory=/var/lib/etcd-fio-test --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300
rm -r /var/lib/etcd-fio-test

In the report, read the fdatasync percentile block. The 99th percentile should be below 10 ms. Anything above 20 ms is unsuitable for a control plane.

The original article suggested a 4k random-write test with a queue depth of one. It does measure latency, but it does not issue a sync after each write, so it does not reproduce how Raft writes behave.

2. Check CPU steal

Sample for a while, ideally under load, and watch the st column. Persistent steal is a hypervisor problem that no orchestrator setting fixes.

vmstat 1 30
top

3. Watch etcd's own fsync metric on a running cluster

On a kubeadm-style control plane node, the metrics endpoint can be queried with the etcd client certificates. Check the histogram etcd_disk_wal_fsync_duration_seconds, and watch etcd_server_leader_changes_seen_total for repeated leader changes.

curl -s --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://127.0.0.1:2379/metrics | grep -E 'wal_fsync_duration|leader_changes'

4. Optional: raise etcd I/O priority and widen the timing margin

Only after the disk has been moved or confirmed adequate. Give the running process the best-effort class at top priority:

ionice -c2 -n0 -p $(pgrep -x etcd)

For timeouts, add flags to the etcd command (in a kubeadm cluster, the static pod manifest at /etc/kubernetes/manifests/etcd.yaml). The values here are illustrative and must be chosen from measured latency, with the election timeout kept well above the heartbeat interval.

--heartbeat-interval=250
--election-timeout=2500

5. Host settings for any orchestrator

The original sysctl list contained settings that either do nothing or fail. net.ipv4.tcp_window_scaling is already on by default. The conntrack key only exists after the nf_conntrack module is loaded. Put the settings in a drop-in file instead of editing the main file. The size values below are examples, to be adjusted to the expected number of tracked connections.

modprobe nf_conntrack
cat > /etc/sysctl.d/90-orchestrator.conf <<'EOF'
net.ipv4.ip_forward = 1
net.netfilter.nf_conntrack_max = 131072
EOF
sysctl --system

IP forwarding is required for container networking on all three orchestrators. Raising rmem_max and wmem_max is not needed for a control plane and can be skipped.

6. Swarm: initialise with an odd number of managers

Run on the first node, then add managers and workers with the join tokens that the command prints. Three managers tolerate the loss of one. Open the swarm ports (2377/tcp, 7946/tcp and udp, 4789/udp) between nodes only.

docker swarm init --advertise-addr 192.168.10.5
docker swarm join-token manager
docker swarm join-token worker

Repeat the fio test on every manager. Manager nodes keep the Raft log under the Docker data directory (/var/lib/docker/swarm by default), so that is the filesystem to test.

Looking back

What works. Treating the control plane disk as a measured requirement removes the most common cause of flapping clusters. A short fio run and a look at st before installing anything catches both a slow disk and an oversold CPU.

Limits.

  • The 10 ms and 20 ms thresholds apply to the 99th percentile on an otherwise idle device. A disk that passes alone can fail when a database shares it.
  • Steal time and noisy neighbours depend on the host, so check again under realistic load and not only right after provisioning.
  • Raising timeouts trades failover speed for tolerance. It does not repair a slow disk.
  • Claims such as lower mesh latency for Swarm than for Kubernetes network plugins, or a given node count for one Nomad server, are not established by the sources above and should be tested rather than assumed.

When to choose another method. If a team cannot operate a three-node control plane, a managed control plane, or a Swarm or Nomad setup with a smaller footprint, may be a better fit than tuning Kubernetes. Nomad suits mixed workloads that are not all containers. Kubernetes suits teams that need its ecosystem and can afford the operational load. Legal questions about where data is stored and who operates the infrastructure (GDPR, transfer rules) are independent of this technical choice. Settle them separately, with a qualified adviser, before deciding on a hosting model.