GitOps in Production: Stop kubectl apply-ing Yourself to Failure

I still see it happen. A senior engineer opens a terminal, points to the production cluster, and types kubectl apply -f deployment.yaml. It works. The pod restarts. The ticket is closed.

Two weeks later, the node crashes, the cluster autoscales, and suddenly the application reverts to a version from three months ago. Why? because the YAML on the engineer's laptop was never committed to Git, and the cluster state was ephemeral. This is "Config Drift," and it is the silent killer of uptime.

If you are managing infrastructure in 2019 without a Single Source of Truth, you are not doing DevOps. You are doing "HopeOps." In this guide, we are going to architect a GitOps workflow that actually survives contact with reality, specifically tailored for high-compliance environments here in Norway.

The Philosophy: If It's Not in Git, It Doesn't Exist

GitOps isn't just a buzzword we stole from Weaveworks; it's an operating model. The core principle is simple: Git is the source of truth. The cluster is merely a reflection of Git.

In a traditional push-based pipeline (like Jenkins running scripts), your CI server holds the "Keys to the Kingdom" (your KUBECONFIG). From a security perspective, this is a nightmare. If someone compromises your Jenkins instance, they own your production environment.

With a Pull-based GitOps approach, the cluster reaches out to the repository. The credentials stay inside the cluster. This is critical for meeting strict ISO 27001 and GDPR requirements, especially when dealing with sensitive Norwegian consumer data under the oversight of Datatilsynet.

The Architecture

We are going to use the following stack, standardizing on tools proven stable as of early 2019:

• VCS: GitLab (or GitHub)
• CI: GitLab CI (for building images)
• CD Agent: Weave Flux (running inside the cluster)
• Infrastructure: CoolVDS KVM instances (High I/O is non-negotiable here)

1. The Application Repository

Your application code lives here. The CI pipeline does only one thing: build the Docker image and push it to a private registry.

Here is a battle-tested Dockerfile structure that keeps layers small. I've seen builds fail because people use massive base images. Stop it. Use Alpine.

FROM node:10-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["node", "server.js"]

2. The Config Repository

This is where the magic happens. You separate your code from your configuration. This repo contains pure YAML manifests. Helm charts are fine, but Tiller (Helm 2's server-side component) introduces massive security holes if not TLS-secured properly. For high-security setups, I often prefer raw manifests or Kustomize.

Here is a standard Deployment manifest. Notice the resource limits. If you don't define these, a memory leak in one pod will OOM-kill your entire node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-processor
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-processor
  template:
    metadata:
      labels:
        app: payment-processor
    spec:
      containers:
      - name: app
        image: registry.coolvds.com/norway-team/payment:v1.4.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10

3. The Synchronization Loop

Instead of your CI script running kubectl set image, it simply commits a change to the Config Repository. It updates the image tag in the YAML file.

Inside your CoolVDS Kubernetes cluster, the Flux operator detects this Git commit. It pulls the new manifest and applies it.

Pro Tip: When using GitOps operators like Flux, network latency creates lag. If your git host is in the US and your servers are in Oslo, you will see sync delays. Hosting your GitLab instance on a CoolVDS slice in the same datacenter as your cluster reduces sync detection to milliseconds.

Implementing the Pipeline

Here is a snippet for your .gitlab-ci.yml. Note that we are not deploying here. We are only updating the manifest file in a separate repo.

stages:
  - build
  - update-manifest

build_image:
  stage: build
  image: docker:stable
  services:
    - docker:dind
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

update_config:
  stage: update-manifest
  image: alpine:3.9
  before_script:
    - apk add --no-cache git openssh-client
    - eval $(ssh-agent -s)
    - echo "$GIT_SSH_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
  script:
    - git config --global user.email "ci-bot@coolvds.com"
    - git clone git@gitlab.com:org/config-repo.git
    - cd config-repo
    - sed -i "s|image: .*|image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA|" deployment.yaml
    - git commit -am "Update image to $CI_COMMIT_SHA"
    - git push origin master

The Hardware Reality Check

GitOps sounds purely software-based, but it places heavy stress on your control plane. The operator (Flux) is constantly cloning repos, unmarshalling YAML, and diffing against the API server. In a shared hosting environment with "noisy neighbors," I have seen the API server latency spike, causing the operator to timeout.

This is why we use CoolVDS. You aren't fighting for CPU cycles in a congested container. You get a KVM slice with dedicated NVMe storage. When you are running etcd (the brain of Kubernetes), disk write latency is the single biggest bottleneck.

Why This Matters for Norway

We operate in a jurisdiction with strict data sovereignty laws. If you push code that processes Norwegian PII (Personally Identifiable Information), you need to know exactly where that processing happens. By hosting your GitOps runners and your production clusters on CoolVDS in Oslo, you ensure that data never leaves the border unnecessarily.

Furthermore, local peering via NIX ensures your customers in Bergen or Trondheim aren't routing through Stockholm to get to your service. Speed is a feature.

Final Thoughts

Manual server management is dead. It died the moment containers became mainstream. Adopting GitOps might feel like extra overhead initially, but the first time you have to rollback a bad deploy by simply running git revert, you will understand.

Don't build a modern workflow on obsolete hardware. Your orchestration layer needs room to breathe. Deploy a CoolVDS KVM instance today and give your Kubernetes cluster the foundation it deserves.