Console Login

K3s, Kubernetes or Nomad: Check Disk Latency Before Choosing

The problem

A small team wants container orchestration on a handful of virtual servers and has three realistic candidates: upstream Kubernetes, K3s (a certified, single-binary Kubernetes distribution) and HashiCorp Nomad. The usual comparison is about features and memory use. The failures that hurt in practice are different. They look like this:

  • The API server answers slowly or times out, and kubectl commands hang intermittently.
  • The datastore logs warnings that applying entries took too long, or that heartbeats were missed.
  • The cluster loses its leader repeatedly, so scheduling stalls even though the workloads themselves are healthy.
  • The control plane competes with the application for CPU and RAM on the same small machine.

Choosing a lighter orchestrator reduces the memory part of this. It does not remove the disk part, which is the same for every candidate that keeps replicated state.

Why it happens

Kubernetes keeps cluster state in etcd. K3s uses SQLite (through the Kine adapter) on a single server by default, or embedded etcd when started in cluster mode. Nomad servers keep state with the Raft consensus protocol. In a Raft-based store, every write must be flushed to disk (fsync) on the leader, replicated, and flushed again on the followers before it is acknowledged.

Disk latency is therefore part of leader liveness. If a flush takes longer than the heartbeat and election timeouts allow, followers assume the leader is gone and start an election. Typical triggers are:

  • Shared or network-attached storage with unpredictable latency.
  • Another process writing heavily to the same disk, such as logs, image pulls or a database.
  • A control plane squeezed into too little CPU or RAM, so it is slow even when the disk is fine.

Source of truth

The fix relies on the upstream documentation of the datastore and the distribution:

  • etcd FAQ confirms that etcd performance depends strongly on disk performance and that fio is a suitable way to check a disk. It states that losing a leader because of disk latency spikes is intentional, because disk latency is part of leader liveness. It gives a target of a 99th percentile wal_fsync_duration_seconds below 10 ms, and says that contention with other applications or a shared virtualized disk is a common cause of slow applies.
  • etcd tuning guide confirms the default heartbeat interval of 100 ms. It says the election timeout should be derived from the heartbeat interval and the round-trip time between members, and that slow disks can cause missed heartbeats, request timeouts and temporary leader loss.
  • K3s requirements confirms a minimum of 2 cores and 2 GB RAM for a server node, excluding the workload. It states that K3s performance depends on the performance of the database, and that etcd is write intensive.
  • K3s resource profiling confirms that a single-node K3s server running a small workload needed roughly 1.6 GB of RAM in the published test, with SQLite and embedded etcd close to each other. This contradicts the common claim that K3s idles at around 600 MB once a workload is present.

Fix options

Option 1: Faster, isolated local storage

Put the control-plane data directory on a local SSD or NVMe volume that is not shared with noisy workloads. This addresses the root cause for every orchestrator and needs no configuration changes.

Option 2: Loosen the timeouts

Raise the etcd heartbeat interval and election timeout so that occasional slow flushes no longer trigger elections. This hides symptoms and makes real leader failures take longer to detect, so it is a stopgap.

Option 3: Use a lighter control plane

  • K3s, single server: SQLite avoids Raft entirely. There is no automatic failover, so it suits small environments where a restore from backup is acceptable.
  • K3s, embedded etcd: start the first server with --cluster-init and run an odd number of servers (three is the usual minimum for high availability).
  • Nomad: a scheduler in one binary that uses HCL job files. It can run Docker containers and also plain executables, which helps with legacy binaries. Servers still use Raft, so disk latency still matters. Nomad has built-in service registration since version 1.3, so Consul is optional.
  • Upstream Kubernetes: the full component set (kube-apiserver, kube-controller-manager, kube-scheduler, etcd) with the widest ecosystem, and the most operating effort.
AspectKubernetesK3sNomad
State storeetcdSQLite or embedded etcdRaft (built in)
Disk-latency sensitivityHighDepends on datastoreHigh on servers
Definition formatYAML manifestsYAML manifestsHCL job files
Non-container workloadsNot nativeNot nativeSupported by task drivers
Operating effortHighestModerateLowest of the three

Why this fix

The disk comes first, because it is the shared root cause and the cheapest to verify. Measure the storage before installing anything, and treat timeout tuning as a fallback only.

For the orchestrator, a small team that wants the Kubernetes API on a few virtual servers is best served by K3s. It keeps the standard API and tooling, the install is one script, and the default SQLite store removes consensus traffic until high availability is actually needed. Nomad is the better pick when the workloads are mixed or when YAML-heavy Kubernetes tooling is not wanted. Upstream Kubernetes fits when the team needs its full ecosystem, for example service meshes or elaborate RBAC, and has people to run it.

Memory planning should follow the published figures: 2 GB is the minimum for a server node, and a server that also hosts a workload needs noticeably more than that.

Step by step

The commands below are instructions to apply on a fresh Linux server. No output is reproduced here, because none of these commands was executed for this article. Judge results against the thresholds from the etcd documentation.

1. Prepare the kernel and swap

The kubelet refuses to start with swap enabled by default, and setting vm.swappiness to 0 does not turn swap off. Disable swap itself and remove its line from /etc/fstab.

swapoff -a
sed -i '/\sswap\s/ s/^/#/' /etc/fstab
modprobe overlay
modprobe br_netfilter
printf 'overlay\nbr_netfilter\n' > /etc/modules-load.d/k8s.conf
# /etc/sysctl.d/99-k8s-networking.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.netfilter.nf_conntrack_max = 131072
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
sysctl --system

The inotify limits matter for log shippers that tail many container log files. When the limits are too low, they can stop following files.

2. Measure the disk

Run the fsync benchmark on the filesystem that will hold the datastore, and read the fdatasync latency percentiles in its report. The 99th percentile should stay below 10 ms.

mkdir -p /var/lib/rancher/fio-test
fio --name=etcd-fsync --directory=/var/lib/rancher/fio-test --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300
rm -r /var/lib/rancher/fio-test

3. Install the K3s server

Download and read the install script before running it, and pin a release channel instead of tracking the newest version. The Flannel VXLAN backend is already the default, so no extra flag is needed. Avoid --write-kubeconfig-mode 644 on shared machines, because it makes the cluster-admin kubeconfig readable by every local user.

curl -sfL https://get.k3s.io -o k3s-install.sh
less k3s-install.sh
INSTALL_K3S_CHANNEL=v1.30 INSTALL_K3S_EXEC="server --disable traefik --node-name cp-01" sh k3s-install.sh

For embedded etcd, which is required for a multi-server setup, add --cluster-init to the first server's arguments. Optional settings go into the K3s configuration file; the values below are examples to adapt after measuring, not recommendations.

# /etc/rancher/k3s/config.yaml
cluster-init: true
etcd-expose-metrics: true
etcd-arg:
  - 'heartbeat-interval=200'
  - 'election-timeout=2000'

With metrics exposed, watch the WAL fsync duration histogram and the leader-change counter that etcd publishes. A p99 above 10 ms or a rising leader-change count points back to the disk.

4. Verify the server and join an agent

systemctl status k3s
k3s kubectl get nodes
cat /var/lib/rancher/k3s/server/node-token

On the worker, point K3S_URL at the server's private address on port 6443 and pass the token read above.

curl -sfL https://get.k3s.io | K3S_URL=https://10.0.0.5:6443 K3S_TOKEN=<node-token> INSTALL_K3S_CHANNEL=v1.30 INSTALL_K3S_EXEC="agent --node-name worker-01" sh -

5. Ingress without deprecated headers

The ingress-nginx controller supports body-size and timeout annotations directly. Snippet annotations are disabled by default in recent controller releases, and the X-XSS-Protection header is obsolete in modern browsers. Set response headers globally through the controller's ConfigMap instead of per-Ingress snippets.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: storefront-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: '50m'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '120'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '120'
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - shop.example.no
    secretName: tls-secret
  rules:
  - host: shop.example.no
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

Alternative: a Nomad job

If Nomad was chosen instead, a cache service looks like this, using Nomad's own service registration rather than Consul:

job "redis-cache" {
  datacenters = ["dc1"]
  type        = "service"

  group "cache" {
    count = 1

    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"

      config {
        image = "redis:7.2-alpine"
        ports = ["db"]
      }

      resources {
        cpu    = 500
        memory = 256
      }

      service {
        name     = "redis-cache"
        port     = "db"
        provider = "nomad"
      }
    }
  }
}
nomad job run redis-cache.nomad.hcl

Looking back

What the approach covers. Measuring fsync latency first, then picking the smallest control plane that fits, addresses the cause rather than the symptoms. The K3s steps give a standards-compliant Kubernetes API with a short install path.

Limits.

  • None of the commands above was run for this article, so exact paths and flags should be checked against the installed K3s release.
  • A single K3s server with SQLite has no failover. Take regular backups of the data directory or the datastore snapshot.
  • Loosening etcd timeouts is a stopgap that delays detection of real failures.
  • Resource needs scale with the number of agents and workloads. Use the sizing tables in the K3s documentation rather than rules of thumb.
  • Any claim about legal or data-residency benefits depends on where the servers are located and on the provider's terms, not on the orchestrator.

When to choose differently. Pick Nomad for mixed container and non-container workloads and for teams that want fewer moving parts. Pick upstream Kubernetes when the ecosystem or a large team justifies its operating cost. Pick K3s with embedded etcd on three servers when automatic control-plane failover is required, and run the disk benchmark on every one of them.