Console Login

Hardening Docker and Kubernetes Containers After Log4Shell (2022)

The problem

Log4Shell (CVE-2021-44228) is remote code execution in the Log4j 2 logging library. It showed how little stands between a flaw in one library and a full foothold when a container runs with its defaults. Once an attacker can run code inside the application process, a default container usually allows the following:

  • The process runs as root (UID 0) inside the container.
  • The root filesystem is writable, so tools, miners or modified configuration can be dropped and kept.
  • The process holds a set of Linux capabilities it never uses.
  • In Kubernetes, every pod can open connections to every other pod, so a compromised frontend can reach a database directly.

Patching the vulnerable library is the first step. It does not limit what the next zero-day can do after it lands in the same kind of container. The aim here is to shrink that blast radius.

Why it happens

Container defaults favour compatibility over containment.

  • Root user. An image without a USER instruction starts its process as UID 0. Unless user namespaces (userns-remap) or rootless mode are enabled, UID 0 in the container is also UID 0 on the host, restricted only by capabilities, seccomp and any AppArmor or SELinux profile. A runtime bug such as CVE-2019-5736, where a malicious process could overwrite the host runc binary, turns that into host root.
  • Writable layer. Docker adds a writable layer on top of the image unless told otherwise.
  • Capabilities. Docker starts containers with a default set of capabilities. A common misconception is that this set includes powerful ones such as CAP_SYS_MODULE or CAP_SYS_TIME. It does not, and those are only available with explicit --cap-add or --privileged. The default set still contains NET_RAW, DAC_OVERRIDE, SETUID, SETGID and CHOWN, which most services do not need.
  • Flat pod network. A Kubernetes pod is non-isolated until a NetworkPolicy selects it, so by default all ingress and egress is allowed.
  • Shared kernel. Containers on one node share one kernel. A kernel or runtime vulnerability crosses container boundaries, and a kernel panic takes down every container on the node.

Source of truth

The fixes below rely on these references:

  • Docker run reference. It documents --read-only, --tmpfs, --user, --cap-drop and --cap-add. It also lists the capabilities a container gets by default and shows that --cap-drop=ALL followed by selective --cap-add is the supported way to restrict them.
  • Dockerfile reference, USER. It confirms that USER sets the user for the rest of the build stage and for the container's default process.
  • Kubernetes network policies. They confirm that pods are non-isolated by default and become isolated once a policy selects them. They also confirm that policies only take effect if the cluster's network plugin enforces them, and they show the default-deny pattern.
  • Kubernetes security context task. It documents runAsNonRoot, runAsUser, readOnlyRootFilesystem, allowPrivilegeEscalation, capabilities and seccompProfile.
  • NVD entry for CVE-2019-5736. It describes the runc container escape and the fixed versions.
  • NVD entry for CVE-2021-44228. It describes the Log4j 2 flaw behind Log4Shell.

Fix options

AreaOptionTrade-off
Process userUSER in the DockerfileTravels with the image, simple. Files must be readable by that user.
Process user--user or Kubernetes runAsNonRootEnforced at deploy time even for images that forgot USER.
Process userUser namespaces or rootless DockerMaps container root to an unprivileged host UID. Stronger, but has volume and networking limits.
Filesystem--read-only plus --tmpfs or volumesBlocks persistence. Needs the writable paths identified first.
Privileges--cap-drop=ALL plus selective --cap-addLeast privilege. Needs testing, because some images need CHOWN, SETUID or SETGID at start-up.
PrivilegesDrop only selected capabilitiesEasier, but still allows everything not named.
NetworkDefault-deny NetworkPolicy plus allow rulesNeeds a network plugin that enforces policy, and egress rules for DNS.
IsolationSandboxed runtimes (gVisor, Kata) or one VM per trust zoneStrongest kernel boundary, with extra overhead and complexity.

Why this fix

The practical choice is to layer the cheap, well-documented controls. They address the conditions an attacker needs after code execution, and they do not conflict with each other:

  1. A non-root user in the image, enforced again at runtime. This removes the easiest route to a runtime escape and to file modification.
  2. A read-only root filesystem with explicit tmpfs or volume mounts. This removes the easy persistence path.
  3. cap_drop: ALL with only the capabilities the service demonstrably needs, plus no-new-privileges, so setuid binaries cannot raise privileges.
  4. A default-deny NetworkPolicy with explicit allow rules, including DNS, so a compromised pod cannot reach arbitrary peers.

These controls need no new software, only configuration, and they are supported by Docker 20.10 and Kubernetes 1.19 or later. Sandboxed runtimes and per-VM isolation are reserved for workloads where the shared kernel is itself the unacceptable risk.

Step by step

The commands below are meant to be applied and checked on a test host. This article does not reproduce terminal output. The expected result of each check is described in prose.

1. Non-root image

The official Node.js images already ship an unprivileged node user (UID 1000). Using it avoids creating one. Leaving application files owned by root keeps them read-only to the process, which is better than a recursive chown to the app user.

FROM node:16-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
USER node
CMD ["node", "index.js"]

For images without a ready-made user, create one with addgroup -S appgroup && adduser -S appuser -G appgroup (Alpine) before the USER line.

2. Run with a read-only filesystem and no capabilities

docker build -t my-secure-image:latest .
docker run -d --name app \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -p 8080:8080 \
  my-secure-image:latest

Port 8080 is above 1023, so NET_BIND_SERVICE is not needed. A service that binds port 80 or 443 inside the container needs --cap-add NET_BIND_SERVICE. Stock nginx images also typically need CHOWN, SETGID and SETUID because the master process starts as root and drops to the worker user. Add them back one at a time if start-up fails with a permission error. Prefer writing logs to stdout and stderr, which avoids a writable log volume.

The same settings in Compose file format 3.x:

version: '3.8'
services:
  app:
    image: my-secure-image:latest
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    ports:
      - "8080:8080"

3. Verify

docker exec app id
docker exec app touch /app/test.txt
docker exec app touch /tmp/test.txt
docker inspect --format '{{.HostConfig.ReadonlyRootfs}} {{.HostConfig.CapDrop}}' app

id should report UID 1000 and not 0. The write to /app should fail with a read-only file system error. The write to /tmp should succeed, because it is a tmpfs. The inspect command should show the read-only flag as true and ALL in the dropped capabilities. To find what a service needs, start it with --cap-drop ALL in a test environment and watch for "Operation not permitted" errors in its logs. capsh --print inside a container lists the capability sets it actually holds.

4. The same controls in Kubernetes

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: my-secure-image:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]
    volumeMounts:
    - name: tmp
      mountPath: /tmp
  volumes:
  - name: tmp
    emptyDir: {}

5. Default-deny network policy, with DNS and one allowed path

Denying egress also blocks name resolution, so DNS has to be allowed explicitly. The kubernetes.io/metadata.name namespace label is set automatically from Kubernetes 1.21.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

The frontend pods also need an egress rule to the backend, because default-deny covers both directions. Apply and inspect the policies with:

kubectl apply -f policies.yaml
kubectl get networkpolicy -n default
kubectl describe networkpolicy default-deny-all -n default

Then test from a throwaway pod with kubectl exec and a tool such as wget, confirming that the allowed path connects and every other destination times out.

Limits and alternatives

  • Not a replacement for patching. These controls reduce what a compromised process can do. They do not stop the initial exploit. The vulnerable Log4j version still has to be upgraded.
  • Kernel and runtime bugs remain. A non-root user and dropped capabilities make many escapes harder but not impossible. Keep the container runtime and kernel updated, since fixes such as the one for CVE-2019-5736 ship there.
  • Read-only does not cover memory. Code can still run in memory, and anything written to a tmpfs or volume stays writable. Mount tmpfs with noexec where the application allows it.
  • Policies need enforcement. A NetworkPolicy object does nothing on a cluster whose network plugin ignores it. Check that the plugin supports policy before relying on it.
  • Capability tuning costs effort. Some images only start with extra capabilities, and overly broad egress allow rules weaken the default-deny.
  • Cluster-wide enforcement. PodSecurityPolicy is deprecated as of Kubernetes 1.21. The built-in Pod Security admission arrives as a beta in 1.23, and third-party admission controllers can enforce the same rules until then.
  • Stronger isolation. For untrusted or multi-tenant workloads, use rootless mode or user namespaces, a sandboxed runtime, or separate virtual machines. A hardware-virtualised VM gives each trust zone its own kernel, which containers on one node do not have.