Edge MQTT Node on a Norwegian VPS: K3s, WireGuard, Kernel Tuning

The problem

Sensors, vessels and gateways in Norway often send telemetry to a cloud region in another country. Every MQTT session pays for that distance several times: the TCP handshake, the TLS handshake and the MQTT CONNECT each cost at least one round trip. On unstable mobile or satellite links, long sessions are also more likely to drop and reconnect.

The usual answer is a small edge aggregator. It is a node close to the data sources that accepts the MQTT traffic, filters and batches it, and forwards archives to a central system. Edge guides for this pattern tend to share the same defects when followed literally:

• The K3s install uses a flag (--no-deploy) that is deprecated and rejected by recent releases.
• The WireGuard interface is enabled with a systemd unit name that does not exist.
• The Deployment references a PersistentVolumeClaim and a ConfigMap that are never defined.
• The sysctl block contains an option that does not do what its comment says.
• A single replica is described as high availability.

Why it happens

Latency

Signals in optical fibre travel at roughly two thirds of the speed of light in vacuum, and every router hop adds queuing and processing time. Round-trip time therefore grows with route length, and no amount of bandwidth removes that. A node in the same country as the clients removes most of the long-haul part of the path. A last-mile link such as satellite, however, keeps its own latency. A sub-5 ms figure can only apply to the terrestrial segment between a client and the node, and it has to be measured on the real path.

Broken instructions

• K3s flags: packaged components are now disabled with --disable or a config file entry. The older --no-deploy spelling was deprecated and has been removed in recent releases.
• WireGuard unit: the Debian and Ubuntu package ships a templated unit, wg-quick@.service. The instance name is the config file name, so the unit is wg-quick@wg0, not wg0.
• Volumes: a Deployment that mounts a missing ConfigMap or PVC stays in ContainerCreating or Pending.
• Mosquitto 2.x: since 2.0 the broker listens only on localhost unless a listener is configured, and a configured listener denies anonymous clients unless authentication is set up. An image started without a config shows exactly this behaviour.
• sysctl: fs.file-max is a system-wide ceiling, not a per-process limit. net.ipv4.tcp_tw_reuse only affects outgoing connections, so it does nothing for a broker that accepts them. The old tcp_tw_recycle option, which is the real "fast recycling" knob, was removed from Linux in kernel 4.12 because it broke clients behind NAT.
• Persistence: sysctl -w changes are lost on reboot.
• High availability: one replica on one host is not highly available. It is a single point of failure with automatic restart.

Source of truth

The K3s documentation page Managing Packaged Components is the authoritative reference for the install flag. It confirms the following:

• K3s ships packaged components deployed as add-ons: coredns, traefik, local-storage and metrics-server.
• Those components can be switched off with the --disable flag, for example --disable=traefik, either on a new cluster or on an existing one.
• An empty traefik.yaml.skip file in the manifests directory before the first start also stops that manifest from being deployed.
• The packaged local-storage component is what provides the default storage class. A PVC therefore does not need a hand-written StorageClass on a single node.

The remaining points in this article (WireGuard unit naming, Mosquitto 2.x listener behaviour, sysctl semantics) come from the upstream manual pages and kernel documentation of each project. Check them against the versions installed on the host with man wg-quick, man mosquitto.conf and the kernel's ip-sysctl documentation.

Ways to fix it

Running the broker

Disabling Traefik in K3s

• Pass --disable=traefik through INSTALL_K3S_EXEC at install time.
• Put disable: traefik in /etc/rancher/k3s/config.yaml, which survives reinstalls and upgrades.
• Create traefik.yaml.skip in the manifests directory.

Protecting administration

• A WireGuard tunnel to the core network, with the Kubernetes API and SSH reachable only over it.
• An SSH tunnel or bastion host, which is simpler but does not give a routed network for node-to-node traffic.
• A public API restricted by source-address allow-lists only. This is the weakest option when client addresses change.

Why this fix

• K3s with a config file. One small binary suits a single VPS. The storage provisioner is included, and the manifests are the same ones a larger cluster would use later. The config file is preferred over an environment variable because it is easy to review and keep in version control. Traefik is disabled because the node is meant to expose raw TCP (MQTT), not HTTP ingress.
• WireGuard for administration. It is part of the mainline Linux kernel (since 5.6) and uses a small configuration. It exposes one UDP port and stays silent to unauthenticated packets. That makes it a better fit than OpenVPN for a node whose only management path should be a private address. OpenVPN remains a valid choice where a client platform lacks WireGuard support.
• Persisted sysctl file and corrected values. A file under /etc/sysctl.d/ survives reboots. The socket-buffer and SYN settings are kept. tcp_tw_reuse is dropped because it has no effect on the accepting side.
• Recreate strategy for a single replica. A local volume can only be mounted by one pod, so a rolling update would try to start the new pod while the old one still holds the volume.

Step by step

The sequence below is a procedure to apply on a fresh Ubuntu 24.04 LTS host. It has not been executed for this article, so check the result of each verification command on the target system.

1. Make IP forwarding persistent and install K3s

echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/80-forward.conf
sudo sysctl --system
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'EOF'
disable:
  - traefik
tls-san:
  - 10.100.0.2
EOF
curl -sfL https://get.k3s.io | sudo sh -
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A

The tls-san entry adds the WireGuard address to the API server certificate, so kubectl works over the tunnel without certificate errors. The final two commands should show a Ready node and no Traefik pod.

2. Set up WireGuard

sudo apt update
sudo apt install -y wireguard
umask 077
wg genkey | sudo tee /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key

Create /etc/wireguard/wg0.conf, replacing the placeholders with the key and endpoint values:

[Interface]
Address = 10.100.0.2/24
PrivateKey = <EDGE_PRIVATE_KEY>
ListenPort = 51820

[Peer]
PublicKey = <CORE_PUBLIC_KEY>
Endpoint = core.example.com:51820
AllowedIPs = 10.100.0.0/24
PersistentKeepalive = 25

The MTU line from the original is omitted. WireGuard's default is suitable for a normal 1500-byte path, so lower it only after finding a path with a smaller MTU. PersistentKeepalive matters on the side that sits behind NAT or a stateful modem. The core peer needs the matching entry with the edge public key and AllowedIPs = 10.100.0.2/32.

sudo chmod 600 /etc/wireguard/wg0.conf
sudo systemctl enable --now wg-quick@wg0
sudo wg show
ping -c 3 10.100.0.1

3. Restrict the management ports to the tunnel

Keep an out-of-band console available before enabling a firewall, to avoid locking yourself out.

sudo ufw default deny incoming
sudo ufw allow 51820/udp
sudo ufw allow in on wg0 to any port 22 proto tcp
sudo ufw allow in on wg0 to any port 6443 proto tcp
sudo ufw allow in on wg0 to any port 1883 proto tcp
sudo ufw allow from 10.42.0.0/16
sudo ufw allow from 10.43.0.0/16
sudo ufw enable

The last two rules allow the default K3s pod and service ranges, because a host firewall can otherwise interfere with pod networking. From a machine outside the tunnel, nc -vz <edge-address> 6443 should fail. Over the tunnel it should succeed.

4. Create credentials and deploy Mosquitto

sudo apt install -y mosquitto
sudo systemctl disable --now mosquitto
mosquitto_passwd -c passwd sensor01
sudo k3s kubectl create namespace iot-edge
sudo k3s kubectl -n iot-edge create secret generic mosquitto-passwd --from-file=passwd

Save the following as edge-mqtt.yaml. It defines the ConfigMap, the PVC (using the default storage class from the packaged local-storage component), the Deployment and the Service:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mosquitto-config
  namespace: iot-edge
data:
  mosquitto.conf: |
    persistence true
    persistence_location /mosquitto/data/
    listener 1883
    allow_anonymous false
    password_file /mosquitto/secret/passwd
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mqtt-pvc-nvme
  namespace: iot-edge
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mqtt-edge-node
  namespace: iot-edge
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      securityContext:
        fsGroup: 1883
      containers:
        - name: mosquitto
          image: eclipse-mosquitto:2.0
          ports:
            - containerPort: 1883
              name: mqtt
          volumeMounts:
            - name: mosquitto-config
              mountPath: /mosquitto/config/mosquitto.conf
              subPath: mosquitto.conf
            - name: mosquitto-secret
              mountPath: /mosquitto/secret
              readOnly: true
            - name: mosquitto-data
              mountPath: /mosquitto/data
      volumes:
        - name: mosquitto-config
          configMap:
            name: mosquitto-config
        - name: mosquitto-secret
          secret:
            secretName: mosquitto-passwd
            defaultMode: 0440
        - name: mosquitto-data
          persistentVolumeClaim:
            claimName: mqtt-pvc-nvme
---
apiVersion: v1
kind: Service
metadata:
  name: mqtt-service
  namespace: iot-edge
spec:
  type: LoadBalancer
  selector:
    app: mosquitto
  ports:
    - protocol: TCP
      port: 1883
      targetPort: 1883

sudo k3s kubectl apply -f edge-mqtt.yaml
sudo k3s kubectl -n iot-edge get pods,svc,pvc
sudo k3s kubectl -n iot-edge logs deploy/mqtt-edge-node
mosquitto_pub -h 10.100.0.2 -p 1883 -u sensor01 -P <PASSWORD> -t test/ping -m hello

The pod should reach Running and the PVC should be Bound. The Service type LoadBalancer relies on the bundled K3s service load balancer on a single node. Use NodePort if that component has been disabled.

This listener is plain MQTT, reachable only over the tunnel because of the firewall rules above. Before accepting sensors from the public internet, add a separate TLS listener (certfile, keyfile and the CA settings described in man mosquitto.conf) and open only that port.

5. Tune the kernel

sudo tee /etc/sysctl.d/90-edge.conf >/dev/null <<'EOF'
fs.file-max = 2097152
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 4096
EOF
sudo sysctl --system
sysctl net.ipv4.tcp_syncookies fs.file-max
systemctl show k3s -p LimitNOFILE
ulimit -n

sysctl --system reads /etc/sysctl.d/, whereas a bare sysctl -p reads only /etc/sysctl.conf. The per-process file descriptor limit is what a broker with thousands of clients actually hits. It is set by the service manager (LimitNOFILE), not by fs.file-max, so check the value reported for the K3s unit.

6. Measure the real path

ping -c 20 <edge-address>
mtr -rwc 50 <edge-address>

Run these from the location of the real clients, such as an office, a modem or a vessel gateway. Latency and jitter should be judged from that vantage point and not from the server side.

Limits and alternatives

• What the procedure delivers. It gives a reproducible single-node broker with persistent storage, a private administration path and kernel settings that survive reboots. It does not by itself produce any particular latency. The result depends on the client network and the peering of the hosting location.
• No high availability. A single replica on a single host means a node failure interrupts ingestion. For continuity, run several nodes with a clustered broker or bridge brokers, and accept that this adds real operational complexity. If that is not needed, plain Mosquitto under systemd is easier to run.
• Local storage. The default provisioner uses a directory on the node's disk. Write throughput depends on the underlying disk, so verify it with a tool such as fio before promising ingestion rates.
• Data residency. Placing the server in Norway is one factor in meeting GDPR and Schrems II obligations. The provider's ownership, the data flows to the core system and the contracts also matter. Treat the legal assessment as separate from the technical setup.
• Serverless alternatives. Functions at a CDN edge suit short, stateless HTTP work. They do not fit long-lived MQTT sessions, and cold starts can matter for strict control loops.
• When to choose another method. Use the distribution package when the node only needs a broker. Use a larger cluster when several services must be scheduled across hosts. Use an SSH bastion if a routed private network is not required.