Kubernetes Networking is Broken by Default: A Deep Dive for Northern Europe

I still remember the Tuesday night I almost threw my laptop out the window. We were debugging a microservices cluster for a fintech client in Oslo. Intermittent 503 errors, 400ms latency spikes on internal calls, and zero logs explaining why. It wasn't the code. It wasn't the database. It was iptables choking on 5,000 services.

Most tutorials lie to you. They tell you Kubernetes networking is magic. You apply a YAML file, and traffic flows. In reality, the default networking stack in Kubernetes is a leaky abstraction built on legacy Linux primitives that were never designed for container churn. If you are running production workloads in 2024 without tuning your CNI (Container Network Interface), you are effectively driving a Ferrari with the handbrake on.

This isn't just about throughput. In the Nordic market, where data sovereignty (hello, Datatilsynet) and millisecond-latency to NIX (Norwegian Internet Exchange) are competitive advantages, a sloppy network config can kill your business logic.

The CNI Battlefield: iptables vs. eBPF

For years, Flannel and Calico were the go-to standards. They work. But they rely heavily on iptables. Every time a Service is created, the kernel has to update a massive list of rules. At scale, this operation is O(N). It's slow.

By late 2024, if you aren't looking at eBPF (Extended Berkeley Packet Filter), you're behind. We primarily deploy Cilium for high-performance clusters on CoolVDS. Why? Because it bypasses the iptables spaghetti entirely, handling routing directly in the kernel.

Pro Tip: If you are migrating a legacy cluster, don't just rip and replace the CNI. Test the MTU (Maximum Transmission Unit) settings first. A mismatch between your overlay network and the underlying VPS interface (usually 1500 bytes, but sometimes 9000 for Jumbo Frames) causes packet fragmentation that destroys performance.

Deploying Cilium with Strict Mode

Here is how we initialize a cluster to avoid the kube-proxy overhead entirely. This configuration assumes you are running on a kernel 5.10+ (which is standard on our CoolVDS KVM instances).

This replaces kube-proxy. The latency difference is measurable, especially when your pod density increases.

The Death of Ingress, The Rise of Gateway API

For the longest time, the Ingress resource was the standard. It was simple, but limited. As of late 2024, the Gateway API has matured into the de facto standard for complex traffic routing. It splits the role of the infrastructure provider (CoolVDS/Ops) from the developer.

Here is a practical example. Instead of a monolithic Ingress file, we define a Gateway Class that maps to our load balancer.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cool-gateway
  namespace: production
spec:
  gatewayClassName: cilium
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.coolvds-client.no"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wild-cert

Then, the developer just attaches a route to it. This separation prevents junior devs from accidentally nuking the load balancer config for the entire cluster.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
  - name: cool-gateway
  hostnames:
  - "api.coolvds-client.no"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v2
    backendRefs:
    - name: api-service-v2
      port: 8080

The Physical Layer: Latency and Geography

You can tune software all day, but you cannot beat the speed of light. If your target audience is in Norway, hosting in Frankfurt or Amsterdam adds 15-30ms of round-trip time (RTT). That sounds negligible until you have a microservice architecture where one user request triggers 50 internal API calls. That 20ms RTT compounds into a full second of delay.

We built the CoolVDS network architecture to peer directly at major Nordic exchanges. When you run mtr (My Traceroute) from a local fiber connection in Oslo to our instances, you notice the lack of hops.

Check your current connectivity. Run this on your bastion host:

If you see packet loss on the first hop, your provider is overselling their uplink. No amount of Kubernetes magic will fix a congested switch port.

Security: The Default is "Open"

Kubernetes allows all pod-to-pod traffic by default. In a multi-tenant environment, this is terrifying. If one pod is compromised, the attacker can scan the entire internal network. We mandate the use of NetworkPolicy resources to implement a zero-trust model.

This policy denies all ingress traffic to the database, except from the backend API pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-access-lockdown
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: backend-api
    ports:
    - protocol: TCP
      port: 5432

Applying this requires a CNI that enforces policies (like Calico or Cilium). The default Flannel setup on many budget VPS providers simply ignores these resources, leaving you with a false sense of security.

The Etcd Bottleneck

Kubernetes networking state is stored in etcd. Every time a pod dies or an endpoint changes, etcd writes to disk. If your VPS uses shared spinning rust (HDD) or throttled SSDs, etcd latency spikes. When etcd spikes, the API server times out. When the API server times out, the network controller fails to update routes.

This is why we insist on NVMe storage for Kubernetes control planes. Use fio to verify your disk performance before installing K8s:

If you aren't seeing IOPS in the thousands, your cluster will become unstable under load.

Final Thoughts

Building a robust Kubernetes network requires peeling back the layers of abstraction. It demands a modern CNI using eBPF, a strict adherence to Gateway API standards, and, crucially, underlying hardware that doesn't choke on I/O.

We designed CoolVDS to be the boring, reliable foundation for this complexity. We provide the raw NVMe speed and the low-latency peering to NIX; you bring the configs.

Stop fighting with noisy neighbors and stolen CPU cycles. Deploy a high-performance, Norway-optimized instance on CoolVDS today and see what sub-millisecond internal latency actually feels like.