Console Login

From "ClickOps" to GitOps: Architecting Zero-Downtime Workflows in 2022

From "ClickOps" to GitOps: Architecting Zero-Downtime Workflows in 2022

I still remember the silence in the Slack channel. It was a Friday afternoon in 2019. A junior developer had access to the production cluster context and decided to "fix" a config map manually using kubectl edit. Typo. Save. Crash loop. The entire e-commerce frontend for a major retailer went dark.

That is why we don't do "ClickOps."

If you are managing infrastructure in Norway today, relying on manual intervention is not just sloppy; it is a liability. Between the strict reporting requirements of the Datatilsynet (Norwegian Data Protection Authority) and the unforgiving latency demands of modern users, your state needs to be declarative, versioned, and automated. Enter GitOps.

The Core Philosophy: Git as the Single Source of Truth

GitOps isn't a product; it's a paradigm. The concept is simple: The state of your Git repository must exactly match the state of your production environment. If it's not in Git, it doesn't exist. If someone changes the server manually, an agent (like ArgoCD or Flux) detects the drift and reverts it immediately.

For this architecture, we are focusing on a stack that has matured significantly over the last year:

  • VCS: GitLab (Self-hosted or SaaS, ideally hosted in EU for GDPR/Schrems II compliance).
  • Controller: ArgoCD v2.x.
  • Infrastructure: Kubernetes v1.22+ on KVM-based Virtual Dedicated Servers.
Pro Tip: Never attempt to run a production Kubernetes cluster on OpenVZ or LXC containers. You need kernel-level control for CNI plugins (like Calico or Cilium) and strict resource isolation. This is why at CoolVDS, we strictly provision KVM instances for our VPS Norway offeringsβ€”it prevents the "noisy neighbor" effect from stealing CPU cycles from your control plane.

Step 1: The Directory Structure

A messy repo leads to a messy cluster. In 2022, the industry standard has shifted towards a "monorepo for config, polyrepo for app code" approach. Here is the directory structure I use for high-availability setups:


β”œβ”€β”€ apps/
β”‚   β”œβ”€β”€ base/
β”‚   β”‚   β”œβ”€β”€ deployment.yaml
β”‚   β”‚   β”œβ”€β”€ service.yaml
β”‚   β”‚   └── kustomization.yaml
β”‚   └── overlays/
β”‚       β”œβ”€β”€ production-oslo/
β”‚       β”‚   β”œβ”€β”€ kustomization.yaml
β”‚       β”‚   └── patch-replicas.yaml
β”‚       └── staging/
β”œβ”€β”€ cluster-config/
β”‚   β”œβ”€β”€ namespaces.yaml
β”‚   └── quotas.yaml
└── infrastructure/
    β”œβ”€β”€ ingress-nginx/
    └── cert-manager/

This structure leverages Kustomize. It allows us to keep a DRY (Don't Repeat Yourself) base configuration while applying environment-specific patches. For example, your Oslo production node on CoolVDS might need 5 replicas and High-Availability affinity rules, while staging only needs 1.

Step 2: The CI/CD Split

A common misconception is that your CI pipeline (Jenkins, GitLab CI, GitHub Actions) should deploy to the server. Stop doing this. CI is for integration. CD is for delivery.

Your CI pipeline should do exactly two things:

  1. Run tests.
  2. Build a Docker image and push it to a registry.

Here is a snippet of a clean GitLab CI configuration .gitlab-ci.yml that builds, scans, and updates the manifest repoβ€”but does not touch the cluster:


build_image:
  stage: build
  image: docker:20.10.12
  services:
    - docker:20.10.12-dind
  script:
    - docker build -t registry.example.no/app:$CI_COMMIT_SHA .
    - docker push registry.example.no/app:$CI_COMMIT_SHA

update_manifest:
  stage: deploy
  image: bitnami/git:2.34.1
  script:
    - git clone https://oauth2:${GIT_TOKEN}@gitlab.com/org/infra-repo.git
    - cd infra-repo/apps/overlays/production-oslo
    - kustomize edit set image app=registry.example.no/app:$CI_COMMIT_SHA
    - git commit -am "Bump version to $CI_COMMIT_SHA"
    - git push origin main

Step 3: The ArgoCD Reconciliation

Once the manifest repo is updated, ArgoCD takes over. It runs inside your cluster, pulls the changes, and applies them. This pull-based mechanism is secure by design because your cluster does not need to expose credentials to the outside world; it only needs outbound access to your Git repo.

Here is the Application manifest you apply to your management cluster:


apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nordic-ecommerce-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://gitlab.com/my-org/infra-repo.git'
    targetRevision: HEAD
    path: apps/overlays/production-oslo
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Notice the selfHeal: true flag. If a sysadmin manually deletes a Service, ArgoCD will recreate it instantly. This is the reliability we need.

The Hardware Reality: Why IOPS Matter for GitOps

GitOps involves constant reconciliation loops. The controller (ArgoCD) is frequently querying the Kubernetes API server, which in turn hammers etcd. Etcd is notoriously sensitive to disk latency. If your disk fsync latency exceeds 10ms, your cluster becomes unstable. Leader elections fail. Pods get evicted.

This is where the "Managed Hosting" versus "Commodity VPS" debate ends.

Feature Standard HDD VPS CoolVDS NVMe
Random IOPS ~400 ~100,000+
Latency 5-15ms <0.5ms
Etcd Stability Risk of split-brain Production Grade

When deploying in Norway, specifically in the Oslo region, you also have the advantage of proximity to the NIX (Norwegian Internet Exchange). Combining high-speed NVMe storage with low network latency ensures your GitOps agents detect and sync changes in near real-time.

Handling Secrets (The GDPR Elephant in the Room)

You cannot commit raw secrets to Git. That is a data breach waiting to happen. In 2022, the two leading solutions are HashiCorp Vault (complex) or Bitnami Sealed Secrets (simple).

For most teams, Sealed Secrets is the sweet spot. You encrypt the secret on your laptop using a public key known only to the cluster controller. Even if the repo is public, the data is safe.


# Install the client
brew install kubeseal

# Create a sealed secret
kubectl create secret generic db-pass --from-literal=password=SuperSecret123 --dry-run=client -o yaml > secret.yaml
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

# Now sealed-secret.yaml is safe to commit to Git

Conclusion

GitOps is not just about tools; it is about sleep. It is about knowing that if your data center in Oslo catches fire, you can point your Git repo to a new CoolVDS instance in a different zone, run a bootstrap script, and be back online in minutes, not days.

Don't let legacy workflows slow down your development cycle. Infrastructure should be invisible, fast, and immutable.

Ready to harden your infrastructure? Deploy a KVM-based, NVMe-powered instance on CoolVDS today and get the IOPS your Kubernetes cluster demands.