Symptoms
A typical sequence: a cluster runs with the default open pod network, and a restrictive NetworkPolicy is then applied to a workload, for example a backend API in a production namespace. The policy lists both Ingress and Egress in policyTypes and includes an egress rule meant to permit DNS. Soon afterwards:
- Pods selected by the policy report name-resolution errors such as
Temporary failure in name resolutionori/o timeouton lookups of service names. - Readiness probes that depend on a database or another service start failing, and the pods enter
CrashLoopBackOff. - Lookups work from pods in other namespaces or from pods the policy does not select, and work again once the policy is deleted.
- Connecting by raw IP address to the same destination may still work, which points at DNS and not at the destination.
The pattern (only policy-selected pods break, and only for egress) distinguishes this case from a broken CoreDNS deployment or a broken cross-node overlay.
Root cause
Several separate mistakes tend to be combined in one manifest.
Egress isolation is implicit
A pod is isolated for egress as soon as any policy selects it and lists Egress in policyTypes. From then on only traffic matched by an egress rule is allowed. DNS is ordinary traffic and gets no exemption.
The namespace selector matches a label that does not exist
A namespaceSelector matches namespace labels, not namespace names. A rule such as matchLabels: {name: kube-system} only works if someone has labelled the namespace with name=kube-system. Clusters label every namespace automatically with kubernetes.io/metadata.name, so the selector should use that key. With the wrong key the rule matches nothing, and the DNS rule is silently dead.
Only UDP 53 is allowed
DNS normally uses UDP, but resolvers retry over TCP when a response is truncated, and TCP support is a requirement for DNS implementations (RFC 7766). Cluster DNS services expose both protocols on port 53. A rule that opens only UDP 53 causes intermittent failures for large responses, which are harder to diagnose than a total outage.
Nothing else is allowed either
A policy that selects the backend pods and allows only DNS egress also blocks the backend's connections to its database, cache and other dependencies. The comment "blocks everything else by default" is also misleading: a policy affects only the pods its podSelector picks. Namespace-wide default deny needs podSelector: {}.
What the docs say
- Kubernetes: Network Policies. It confirms that pods are non-isolated until a policy selects them. Once selected, only traffic permitted by the union of the applicable policies is allowed. It also confirms that policies are additive, that
namespaceSelectormatches namespace labels, and thatkubernetes.io/metadata.nameis set automatically. It states that anamespaceSelectorandpodSelectorin the sametoelement are combined with AND, while separate list elements are combined with OR. Finally, it notes that policies take effect only if the network plugin enforces them. - Kubernetes: Debugging DNS Resolution. It describes a test pod with DNS tools, the
k8s-app=kube-dnslabel on the cluster DNS pods (served by CoreDNS), and thekube-dnsService inkube-system. - RFC 7766: DNS Transport over TCP. It confirms that TCP is a required part of DNS transport, which justifies opening TCP 53 as well as UDP 53.
- Cilium: Kubernetes without kube-proxy. This is the reference for the optional Cilium settings mentioned in the review: the API server address must be given explicitly when kube-proxy is replaced.
Fix options
| Option | Advantage | Drawback |
|---|---|---|
Correct the selector: kubernetes.io/metadata.name: kube-system AND k8s-app: kube-dns, ports UDP and TCP 53 | Portable across CNIs, least privilege | Breaks if DNS runs elsewhere or under other labels |
Label the namespace by hand (name=kube-system) to make the original selector work | No policy change | Manual state that is easy to forget on a rebuilt cluster |
Allow port 53 to every namespace (namespaceSelector: {}) | Simple, tolerant of custom DNS placement | Permits DNS to any pod in the cluster |
ipBlock for the cluster DNS Service IP | Independent of labels | Policy is evaluated against the pod IP after service translation in many setups; behaviour varies by plugin |
| CNI-specific policy (for example a DNS-aware Cilium policy) | Fine-grained, can restrict by name | Ties the manifests to one CNI |
Drop Egress from policyTypes | Removes the symptom immediately | Gives up egress control entirely |
If a node-local DNS cache is deployed, pods query a link-local address on the node instead of the cluster DNS pods, and the DNS rule must permit that address instead.
Which fix and why
The first option is the right default: a standard networking.k8s.io/v1 policy that combines the automatic namespace label with the DNS pod label, and opens both UDP and TCP on port 53. It uses only features every policy-enforcing CNI must implement, and it grants the smallest permission that restores DNS.
The DNS rule is best placed in its own namespace-wide policy, next to a default-deny policy. Every pod then gets DNS without repeating the rule in each workload policy, and workload policies only describe their real dependencies. Because policies are additive, the backend policy can then list the database it needs and nothing else.
Step by step
The commands below are the procedure to apply. No terminal output is reproduced here, and nothing in this article claims a particular result was observed.
1. Confirm the label and the DNS pods
kubectl get namespace kube-system --show-labels
kubectl get pods --namespace kube-system -l k8s-app=kube-dns -o wide
kubectl get service --namespace kube-system kube-dns
The namespace listing should include kubernetes.io/metadata.name=kube-system. The service should list port 53 for UDP and TCP.
2. Reproduce the failure from a pod the policy selects
A test pod carrying the same label as the backend is subject to the same policy:
kubectl --namespace production run dnstest --image=registry.k8s.io/e2e-test-images/agnhost:2.39 --labels=app=backend-api --command -- sleep 3600
kubectl --namespace production exec dnstest -- nslookup kubernetes.default
kubectl --namespace production exec dnstest -- nslookup -vc kubernetes.default
The last command forces TCP, which exposes a missing TCP rule. Replace the image tag with one that is available in the cluster's registry if needed.
3. Write the corrected policies
Save as production-policies.yaml. The database rule is an example; replace the label and port with the real dependency.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-policy
namespace: production
spec:
podSelector:
matchLabels:
app: backend-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend-ui
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
The namespaceSelector and podSelector sit in one list element, so they are ANDed: only pods labelled k8s-app=kube-dns in kube-system match. Splitting them into two - items would instead allow every pod in kube-system plus every pod with that label in the policy's own namespace.
4. Apply and retest
kubectl apply -f production-policies.yaml
kubectl --namespace production exec dnstest -- nslookup kubernetes.default
kubectl --namespace production exec dnstest -- nslookup -vc kubernetes.default
kubectl --namespace production describe networkpolicy allow-dns
5. On Cilium, look at drops directly
When Cilium is the CNI and Hubble is enabled, policy verdicts can be inspected without guessing:
hubble observe --namespace production --verdict DROPPED
6. Clean up
kubectl --namespace production delete pod dnstest
Limits and alternatives
- Enforcement depends on the CNI. The API server accepts
NetworkPolicyobjects whether or not anything enforces them. A plugin without policy support ignores them, so a missing symptom is not proof of a working policy. Verify with a deliberate deny test. - Cluster-specific DNS layouts. A node-local DNS cache, a renamed DNS deployment or a different namespace each require adjusting the selector. The labels in step 1 are the reference to check.
- Egress to the API server and other system services is also blocked by default deny. Workloads that talk to the Kubernetes API need their own explicit rule.
- Choose a CNI-specific policy when rules need to be based on DNS names or when L7 filtering is required; the standard object cannot express that.
Related settings from the same topic
Other details commonly found next to this problem in older guides:
- Cilium without kube-proxy. Newer Cilium releases take
trueorfalseforkubeProxyReplacement; the olderstrictvalue was deprecated around 1.14, so check the documentation for the chosen version. The API server address must be given explicitly:
helm install cilium cilium/cilium --version 1.16.0 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=${API_SERVER_IP} \
--set k8sServicePort=${API_SERVER_PORT} \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set ipam.mode=kubernetes
- Ingress class. The
kubernetes.io/ingress.classannotation is the legacy way to pick a controller; current manifests usespec.ingressClassName. - Connection tracking. On busy nodes,
net.netfilter.nf_conntrack_maxandnet.core.somaxconnare the usual sysctls to raise.net.ipv4.tcp_tw_reuseaffects only outgoing connections and should be changed only for a demonstrated need.
# /etc/sysctl.d/99-k8s-networking.conf
net.netfilter.nf_conntrack_max = 1048576
net.core.somaxconn = 65535
net.ipv4.ip_local_port_range = 32768 60999
sysctl --system