Kubernetes Networking in Production: eBPF, Gateway API, and The Latency Trap

I still remember the 3 AM panic call from a fintech client in Oslo last winter. Their payment processing microservices were timing out randomly. The logs were clean, the application code hadn't changed, and the database was sleeping. The culprit? A noisy neighbor on their budget VPS provider was saturating the physical network link, causing micro-burst packet drops that standard monitoring tools missed. We migrated them to dedicated KVM slices, and the latency flatlined immediately.

That war story highlights a truth often ignored in the container hype: Kubernetes is only as stable as the network it runs on.

By now, in May 2025, the "how do I set up a cluster" tutorials are irrelevant. The real challenge is networking at scale—managing east-west traffic, enforcing zero-trust without destroying performance, and handling ingress without drowning in Nginx config spaghetti. If you are running mission-critical workloads in Norway or the broader Nordics, you need to understand the modern networking stack: eBPF, the Gateway API, and the physical reality of packet travel.

The CNI Landscape in 2025: Why eBPF Won

Years ago, we debated iptables versus IPVS. That debate is dead. In 2025, if you aren't using an eBPF-based CNI (Container Network Interface) like Cilium, you are voluntarily accepting performance degradation. Traditional iptables implementations turn into a linear search nightmare when you hit thousands of services. eBPF (Extended Berkeley Packet Filter) allows us to run sandboxed programs in the kernel, processing packets without the context-switching overhead.

In high-throughput environments—like real-time bidding or data ingestion pipelines—this difference is measurable in milliseconds. For a cluster hosted in Oslo serving users in Trondheim or Bergen, that efficiency is critical.

Configuring Cilium for Strict Performance

Don't just install the default Helm chart. You need to enable the native routing features to bypass encapsulation overhead (VXLAN/Geneve) if your underlying network supports it. Here is a production-ready configuration snippet for a high-performance setup:

helm install cilium cilium/cilium --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=API_SERVER_IP \
  --set k8sServicePort=6443 \
  --set bpf.masquerade=true \
  --set tunnel=disabled \
  --set autoDirectNodeRoutes=true \
  --set ipv4.nativeRoutingCIDR=10.0.0.0/8

Pro Tip: Setting tunnel=disabled and autoDirectNodeRoutes=true allows pods to communicate via direct L3 routing. This removes the encapsulation header overhead, increasing MTU efficiency and reducing CPU usage. However, this requires a host network that permits L3 routing between nodes—standard on CoolVDS private networks.

Goodbye Ingress, Hello Gateway API

For years, the Kubernetes Ingress resource was frustratingly vague. It handled HTTP routing but failed miserably at anything else (gRPC, TCP, TLS splitting). By 2025, the Gateway API has matured into the standard for traffic entry.

The Gateway API decouples the infrastructure (the Gateway) from the routing logic (HTTPRoute). This allows your infrastructure team to manage the load balancer listeners while developers own their routing rules.

Here is how a modern routing config looks compared to the old Ingress annotations mess:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payment-service-route
  namespace: payments
spec:
  parentRefs:
  - name: external-gateway
    namespace: infrastructure
  hostnames:
  - "api.payment-system.no"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/charge
    backendRefs:
    - name: charge-service
      port: 8080
      weight: 1
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: X-Region
          value: "NO-OSL-1"

This structure is cleaner, portable, and natively supports advanced traffic splitting (canary deployments) without relying on vendor-specific annotations.

The Hidden Bottleneck: Etcd and Disk I/O

Networking isn't just about moving packets; it's about the control plane knowing where to move them. Kubernetes networking relies heavily on etcd to store the state of the cluster. Every time a Pod starts or a Service updates, etcd writes to disk.

If your disk I/O latency is high (fsync latency > 10ms), etcd will throttle, and your API server will start timing out requests. You will see ApplyLayer failed errors or CNI plugin timeouts.

This is where the infrastructure choice becomes non-negotiable.

I've benchmarked generic "cloud VPS" instances against dedicated NVMe resources. The generic instances often share disk throughput with hundreds of other tenants. When a neighbor decides to re-index a massive database, your K8s control plane stutters.

Benchmarking Your Storage for K8s

Before deploying a cluster, run fio to verify the disk can handle the etcd write load. You need high IOPS and low latency for small, sequential writes.

fio --rw=write --ioengine=sync --fdatasync=1 \
  --directory=test-data --size=22m --bs=2300 \
  --name=etcd-benchmark

On a CoolVDS NVMe instance, the 99th percentile commit latency consistently stays under 2ms. On standard spinning rust or oversold SSDs, I often see spikes over 40ms, which causes leader elections in etcd to fail.

Local Nuances: The Norway Advantage (NIX)

For teams targeting the Norwegian market, physical location is a massive technical advantage. Routing traffic from Oslo to a data center in Frankfurt or Amsterdam adds 15-30ms of round-trip latency. That doesn't sound like much, but in a microservices architecture with multiple internal calls, it compounds.

Hosting locally means your traffic likely hits the NIX (Norwegian Internet Exchange) directly. This keeps data within the country borders—simplifying GDPR compliance with Datatilsynet—and ensures the lowest possible RTT (Round Trip Time) for your local users.

Locking It Down: Network Policies

By default, Kubernetes allows all traffic between all pods. In a multi-tenant environment, this is terrifying. You must implement a NetworkPolicy to restrict traffic. Since we are using Cilium (see above), we can use CiliumNetworkPolicy for L7 filtering (like allowing only HTTP GET requests).

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: secure-backend
  namespace: app
spec:
  endpointSelector:
    matchLabels:
      role: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        role: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET"
          path: "/public/.*"

This policy ensures that even if the frontend is compromised, an attacker cannot use it to send POST or DELETE requests to the backend, nor can they scan other ports. This level of granular control is mandatory for any serious deployment.

The Verdict

Building a robust Kubernetes platform requires peeling back the abstraction layers. You need to control the CNI, understand the API objects like Gateway, and crucially, trust the metal underneath. Software-defined networking is powerful, but it cannot fix a physically congested uplink or a slow disk.

If you are tired of debugging "ghost" latency or worrying about data residency laws, stop fighting the noisy neighbors on oversold clouds. Build your cluster on infrastructure designed for I/O-heavy workloads.

Ready to see what 1ms latency looks like? Deploy a CoolVDS NVMe instance in Oslo today and give your K8s cluster the foundation it deserves.