Console Login

GitOps with Argo CD and Kustomize: Ending Manual kubectl Changes

The problem

A typical production Kubernetes cluster is changed by someone logging in and running kubectl apply -f, or by a CI job that holds cluster credentials and does the same thing. Several problems follow:

  • A hotfix applied by hand leaves no review and no record of who changed what or why.
  • A mistyped command or a wrong kube-context changes the wrong environment.
  • Live objects slowly diverge from the YAML in the repository, so the repository stops describing what is running.
  • Rolling back means remembering what the previous state was.
  • The CI runner needs a credential with broad rights on the cluster. Anyone who compromises the runner or its secrets gets that access.

Raw Kubernetes Secret objects add a second problem. Their values are only base64-encoded, so they cannot be committed to Git as they are. They end up in wikis, shell history or CI variables instead.

Why it happens

The root cause is an imperative, push-based delivery model with no single source of truth.

  • Imperative changes. kubectl apply, kubectl edit and kubectl scale change live state directly. Nothing compares the result with a reviewed definition afterwards.
  • Push credentials. The system that deploys sits outside the cluster, so the cluster API has to accept connections and credentials from it.
  • Mixed concerns. Application source code and deployment manifests live in the same repository. A manifest change then triggers application builds, and release history gets tangled.
  • Per-environment copies. Development and production manifests are copied and edited by hand. The copies drift apart, and a fix applied to one is forgotten in the other.

Source of truth

The behaviour described below is documented by the projects themselves:

  • Argo CD, Automated Sync Policy. It confirms that automated sync applies a change when the desired state in Git differs from the live state. Automated sync does not delete resources that were removed from Git unless prune is enabled. Changes made directly in the cluster are not reverted unless selfHeal is enabled. It also notes that a rollback cannot be started on an application with automated sync enabled. A revert has to go through Git.
  • Argo CD, Best Practices. It recommends keeping the Kubernetes manifests in a separate repository from the application source code. Its reasons are a clean audit log, separate access control and no unwanted build triggers.
  • Sealed Secrets, project repository. It documents the model of encrypting a Secret with the kubeseal client into a SealedSecret resource. That resource is safe to store in a public repository, and only the in-cluster controller can decrypt it. It also documents that the sealed secret is bound to a name and namespace by default.

Fix options

Delivery model

MethodHow it worksDrawback
Push from CIThe pipeline runs kubectl apply or helm upgrade with stored cluster credentials.The CI system holds production credentials. Drift is not detected between runs.
Pull with Argo CDA controller in the cluster watches a Git repository and syncs it. It has a UI and shows a diff of drift.It is one more component to run and secure. The Argo CD API server and UI still need access control.
Pull with Flux v2A set of controllers in the cluster reconciles Git sources. It is configured through Kubernetes custom resources and has no built-in UI.It is less visual. Drift is mostly inspected through the CLI and events.

Secrets in Git

  • Sealed Secrets. Asymmetric encryption with a controller-held private key. It is simple, but the secrets are tied to one cluster's key.
  • SOPS with age or a cloud KMS. It encrypts the values inside any YAML file and works across clusters, but it needs a decryption plugin in the GitOps tool.
  • External secret store with a sync operator. Nothing sensitive is in Git, but a separate secret service has to be operated.

Why this fix

A pull-based Argo CD setup is a good default. It is combined with Kustomize overlays, a separate config repository and Sealed Secrets. The reasons:

  • The cluster pulls from Git, so the CI system no longer needs cluster credentials. It only needs write access to the config repository.
  • Every production change becomes a commit or merge request with a review and a history. A rollback is git revert.
  • selfHeal turns manual edits into temporary events, because the controller reverts them to the Git state.
  • Kustomize ships with kubectl. It keeps one base and small per-environment patches without a template language.
  • Sealed Secrets needs only one controller and one CLI, which suits a small team. SOPS is the alternative when the same secrets must be used in several clusters.

Some claims often attached to this pattern are overstated. Self-healing does not make the cluster immutable. It reverts drift for the resources Argo CD manages. The cluster also does not match Git bit for bit, because the API server adds defaults and controllers mutate some fields. Argo CD compares the rendered manifests with the live state and can be told to ignore chosen fields.

Step by step

This is the procedure. Placeholder host names and registry names must be replaced with real ones.

1. Install Argo CD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deployment/argocd-server

2. Create the config repository

Use a repository separate from the application code, with a base and one overlay per environment:

config-repo/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml
  overlays/
    dev/
      kustomization.yaml
      patch-replicas.yaml
    prod/
      kustomization.yaml
      patch-resources.yaml

The base/kustomization.yaml lists the shared resources:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

The production overlay references the base, patches it and sets the image. The name under images must match the image name used in base/deployment.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
  - sealed-secret.yaml
patches:
  - path: patch-resources.yaml
images:
  - name: my-app
    newName: registry.example.com/app
    newTag: v1.0.0
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi

Render the overlay locally before committing, to catch mistakes early:

kubectl kustomize overlays/prod

3. Register the repository and create the Application

Argo CD needs read access to a private repository. A read-only deploy key is enough:

argocd repo add [email protected]:platform/payments-config.git --ssh-private-key-path ~/.ssh/argocd_deploy_key

Then apply the Application. Pinning targetRevision to a branch name is clearer than HEAD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: '[email protected]:platform/payments-config.git'
    targetRevision: main
    path: overlays/prod
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: payments-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
kubectl apply -f application.yaml

This is the last manual kubectl apply against the cluster for this application. Argo CD checks the repository on a polling interval, which is three minutes by default. A Git webhook to Argo CD makes changes appear sooner.

4. Add secrets safely

Install the Sealed Secrets controller as described in the project's README. Then fetch its public certificate and seal a secret. Reading the password with read keeps it out of shell history. The namespace is given explicitly, because the sealed result is bound to name and namespace:

kubeseal --fetch-cert --controller-name=sealed-secrets-controller --controller-namespace=kube-system > pub-cert.pem
read -rs DB_PASSWORD
kubectl create secret generic db-creds --namespace payments-prod --from-literal=password="$DB_PASSWORD" --dry-run=client -o yaml > secret.yaml
kubeseal --format yaml --cert pub-cert.pem < secret.yaml > overlays/prod/sealed-secret.yaml
rm secret.yaml
unset DB_PASSWORD
git add overlays/prod/sealed-secret.yaml
git commit -m "Add sealed db-creds"

Back up the controller's sealing key. Without it, existing sealed secrets cannot be decrypted after the cluster is rebuilt.

5. Let CI edit only the image tag

The application pipeline runs tests, builds and pushes the image, then changes one line in the config repository. This GitHub Actions job assumes the kustomize binary is available on the runner; add an install step if it is not. It passes the tag through an environment variable rather than pasting it into the script:

name: Update Image Tag
on:
  push:
    tags:
      - 'v*'
jobs:
  update-manifests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout config repo
        uses: actions/checkout@v4
        with:
          repository: my-org/config-repo
          token: ${{ secrets.CONFIG_REPO_TOKEN }}

      - name: Update image tag
        env:
          TAG: ${{ github.ref_name }}
        run: |
          cd overlays/prod
          kustomize edit set image my-app=registry.example.com/app:${TAG}

      - name: Commit and push
        env:
          TAG: ${{ github.ref_name }}
        run: |
          git config user.name "gitops-bot"
          git config user.email "[email protected]"
          git add .
          git commit -m "Update image to ${TAG}"
          git push

The token should be scoped to the config repository only, with write access.

6. Check drift handling and rollback

argocd app get payments-prod
kubectl -n payments-prod scale deployment my-app --replicas=5
kubectl -n payments-prod get deployment my-app --watch

With selfHeal enabled, the controller is expected to restore the replica count that Git defines. A rollback goes through Git:

git revert HEAD
git push

Limits and alternatives

  • Polling delay. Without a webhook, a merged change can take several minutes to appear. Configure the webhook if that matters.
  • Prune is destructive. With prune: true, deleting a manifest from Git deletes the live resource. Test pruning on a non-production application first, and consider stricter settings for stateful workloads.
  • Argo CD is a privileged component. It holds broad rights in the cluster. Restrict access to its API and UI, use SSO or strong credentials, and use Argo CD Projects to limit which repositories and namespaces an application may use.
  • Sealed Secrets is tied to one cluster key. For several clusters sharing a secret, or for key management outside the cluster, SOPS or an external secret store fits better.
  • Hardware. The control plane datastore, etcd, is sensitive to slow disk writes, so slow storage can delay everything on the cluster. Sync time also depends on manifest rendering and the number of applications. Storage is one factor and not a guaranteed fix for slow syncs.
  • Data residency. Where Git hosting and the GitOps controller run can matter for data-protection requirements. Whether a specific setup satisfies a regulation is a legal question that this procedure does not answer.
  • Flux. Choose Flux v2 if a UI is not needed and everything should be handled as Kubernetes custom resources.