Console Login

Kubernetes Pod DNS Fails After an Egress NetworkPolicy: Fix

Symptoms

A typical sequence: a cluster runs with the default open pod network, and a restrictive NetworkPolicy is then applied to a workload, for example a backend API in a production namespace. The policy lists both Ingress and Egress in policyTypes and includes an egress rule meant to permit DNS. Soon afterwards:

  • Pods selected by the policy report name-resolution errors such as Temporary failure in name resolution or i/o timeout on lookups of service names.
  • Readiness probes that depend on a database or another service start failing, and the pods enter CrashLoopBackOff.
  • Lookups work from pods in other namespaces or from pods the policy does not select, and work again once the policy is deleted.
  • Connecting by raw IP address to the same destination may still work, which points at DNS and not at the destination.

The pattern (only policy-selected pods break, and only for egress) distinguishes this case from a broken CoreDNS deployment or a broken cross-node overlay.

Root cause

Several separate mistakes tend to be combined in one manifest.

Egress isolation is implicit

A pod is isolated for egress as soon as any policy selects it and lists Egress in policyTypes. From then on only traffic matched by an egress rule is allowed. DNS is ordinary traffic and gets no exemption.

The namespace selector matches a label that does not exist

A namespaceSelector matches namespace labels, not namespace names. A rule such as matchLabels: {name: kube-system} only works if someone has labelled the namespace with name=kube-system. Clusters label every namespace automatically with kubernetes.io/metadata.name, so the selector should use that key. With the wrong key the rule matches nothing, and the DNS rule is silently dead.

Only UDP 53 is allowed

DNS normally uses UDP, but resolvers retry over TCP when a response is truncated, and TCP support is a requirement for DNS implementations (RFC 7766). Cluster DNS services expose both protocols on port 53. A rule that opens only UDP 53 causes intermittent failures for large responses, which are harder to diagnose than a total outage.

Nothing else is allowed either

A policy that selects the backend pods and allows only DNS egress also blocks the backend's connections to its database, cache and other dependencies. The comment "blocks everything else by default" is also misleading: a policy affects only the pods its podSelector picks. Namespace-wide default deny needs podSelector: {}.

What the docs say

  • Kubernetes: Network Policies. It confirms that pods are non-isolated until a policy selects them. Once selected, only traffic permitted by the union of the applicable policies is allowed. It also confirms that policies are additive, that namespaceSelector matches namespace labels, and that kubernetes.io/metadata.name is set automatically. It states that a namespaceSelector and podSelector in the same to element are combined with AND, while separate list elements are combined with OR. Finally, it notes that policies take effect only if the network plugin enforces them.
  • Kubernetes: Debugging DNS Resolution. It describes a test pod with DNS tools, the k8s-app=kube-dns label on the cluster DNS pods (served by CoreDNS), and the kube-dns Service in kube-system.
  • RFC 7766: DNS Transport over TCP. It confirms that TCP is a required part of DNS transport, which justifies opening TCP 53 as well as UDP 53.
  • Cilium: Kubernetes without kube-proxy. This is the reference for the optional Cilium settings mentioned in the review: the API server address must be given explicitly when kube-proxy is replaced.

Fix options

OptionAdvantageDrawback
Correct the selector: kubernetes.io/metadata.name: kube-system AND k8s-app: kube-dns, ports UDP and TCP 53Portable across CNIs, least privilegeBreaks if DNS runs elsewhere or under other labels
Label the namespace by hand (name=kube-system) to make the original selector workNo policy changeManual state that is easy to forget on a rebuilt cluster
Allow port 53 to every namespace (namespaceSelector: {})Simple, tolerant of custom DNS placementPermits DNS to any pod in the cluster
ipBlock for the cluster DNS Service IPIndependent of labelsPolicy is evaluated against the pod IP after service translation in many setups; behaviour varies by plugin
CNI-specific policy (for example a DNS-aware Cilium policy)Fine-grained, can restrict by nameTies the manifests to one CNI
Drop Egress from policyTypesRemoves the symptom immediatelyGives up egress control entirely

If a node-local DNS cache is deployed, pods query a link-local address on the node instead of the cluster DNS pods, and the DNS rule must permit that address instead.

Which fix and why

The first option is the right default: a standard networking.k8s.io/v1 policy that combines the automatic namespace label with the DNS pod label, and opens both UDP and TCP on port 53. It uses only features every policy-enforcing CNI must implement, and it grants the smallest permission that restores DNS.

The DNS rule is best placed in its own namespace-wide policy, next to a default-deny policy. Every pod then gets DNS without repeating the rule in each workload policy, and workload policies only describe their real dependencies. Because policies are additive, the backend policy can then list the database it needs and nothing else.

Step by step

The commands below are the procedure to apply. No terminal output is reproduced here, and nothing in this article claims a particular result was observed.

1. Confirm the label and the DNS pods

kubectl get namespace kube-system --show-labels
kubectl get pods --namespace kube-system -l k8s-app=kube-dns -o wide
kubectl get service --namespace kube-system kube-dns

The namespace listing should include kubernetes.io/metadata.name=kube-system. The service should list port 53 for UDP and TCP.

2. Reproduce the failure from a pod the policy selects

A test pod carrying the same label as the backend is subject to the same policy:

kubectl --namespace production run dnstest --image=registry.k8s.io/e2e-test-images/agnhost:2.39 --labels=app=backend-api --command -- sleep 3600
kubectl --namespace production exec dnstest -- nslookup kubernetes.default
kubectl --namespace production exec dnstest -- nslookup -vc kubernetes.default

The last command forces TCP, which exposes a missing TCP rule. Replace the image tag with one that is available in the cluster's registry if needed.

3. Write the corrected policies

Save as production-policies.yaml. The database rule is an example; replace the label and port with the real dependency.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend-api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend-ui
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

The namespaceSelector and podSelector sit in one list element, so they are ANDed: only pods labelled k8s-app=kube-dns in kube-system match. Splitting them into two - items would instead allow every pod in kube-system plus every pod with that label in the policy's own namespace.

4. Apply and retest

kubectl apply -f production-policies.yaml
kubectl --namespace production exec dnstest -- nslookup kubernetes.default
kubectl --namespace production exec dnstest -- nslookup -vc kubernetes.default
kubectl --namespace production describe networkpolicy allow-dns

5. On Cilium, look at drops directly

When Cilium is the CNI and Hubble is enabled, policy verdicts can be inspected without guessing:

hubble observe --namespace production --verdict DROPPED

6. Clean up

kubectl --namespace production delete pod dnstest

Limits and alternatives

  • Enforcement depends on the CNI. The API server accepts NetworkPolicy objects whether or not anything enforces them. A plugin without policy support ignores them, so a missing symptom is not proof of a working policy. Verify with a deliberate deny test.
  • Cluster-specific DNS layouts. A node-local DNS cache, a renamed DNS deployment or a different namespace each require adjusting the selector. The labels in step 1 are the reference to check.
  • Egress to the API server and other system services is also blocked by default deny. Workloads that talk to the Kubernetes API need their own explicit rule.
  • Choose a CNI-specific policy when rules need to be based on DNS names or when L7 filtering is required; the standard object cannot express that.

Related settings from the same topic

Other details commonly found next to this problem in older guides:

  • Cilium without kube-proxy. Newer Cilium releases take true or false for kubeProxyReplacement; the older strict value was deprecated around 1.14, so check the documentation for the chosen version. The API server address must be given explicitly:
helm install cilium cilium/cilium --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=${API_SERVER_PORT} \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set ipam.mode=kubernetes
  • Ingress class. The kubernetes.io/ingress.class annotation is the legacy way to pick a controller; current manifests use spec.ingressClassName.
  • Connection tracking. On busy nodes, net.netfilter.nf_conntrack_max and net.core.somaxconn are the usual sysctls to raise. net.ipv4.tcp_tw_reuse affects only outgoing connections and should be changed only for a demonstrated need.
# /etc/sysctl.d/99-k8s-networking.conf
net.netfilter.nf_conntrack_max = 1048576
net.core.somaxconn = 65535
net.ipv4.ip_local_port_range = 32768 60999
sysctl --system