I used to manage deployments across a few Kubernetes clusters with a collection of Helm charts and custom scripts. It was manageable at first, but as we grew to over 15 clusters and hundreds of applications, it became a nightmare. There was no single source of truth, and configuration drift was rampant. That’s when I decided to go all-in on GitOps with ArgoCD. It completely transformed our deployment workflow, and here’s how I did it.
Why I Chose GitOps
The breaking point came during an incident where production behaved differently than staging, even though we’d supposedly deployed the same configuration. After two hours of debugging, I discovered someone had made a manual change to production with kubectl apply to fix an urgent issue three weeks earlier. Nobody documented it. Nobody remembered it. The change wasn’t in Git.
That incident cost us a customer and made me realize our deployment process was fundamentally broken. We needed Git to be the single source of truth. If it’s not in Git, it doesn’t exist. I wanted a system that would look at Git, look at the cluster, and make them match automatically. If someone tried to make a manual change, the system should undo it within minutes.
I evaluated Flux and ArgoCD. Both are solid GitOps tools, but ArgoCD’s UI won me over. Being able to visually see the sync status of hundreds of applications across multiple clusters was invaluable. The built-in webhook support for Git repos meant deployments happened automatically when developers merged pull requests. And the drift detection would catch and fix those manual kubectl apply changes that had burned us.
graph LR
A[Developer] -->|git push| B[Git Repository]
B -->|webhook| C[ArgoCD]
C -->|sync & monitor| D[Kubernetes Cluster]
D -->|detects drift| C
style B fill:#fff4e6
style C fill:#f3e5f5
How I Actually Implemented This
The “App of Apps” Pattern (Or How I Avoided Managing 200+ YAML Files)
My first instinct was to create an ArgoCD Application resource for each of our applications. Then I did the math. We had over 200 applications across our clusters. Managing 200+ YAML files manually sounded like a special kind of hell.
That’s when I discovered the “app of apps” pattern. Instead of managing hundreds of Application resources, I created one root Application that points to a directory in Git. Inside that directory, I put all the individual application definitions. ArgoCD automatically discovers new applications when they’re added to that directory. Onboarding a new app became as simple as adding a YAML file and opening a pull request.
This was a game-changer. A developer could add their application to ArgoCD without ever talking to me. They just followed the template, opened a PR, and once merged, ArgoCD picked it up within 3 minutes.
# My root application that discovers all others
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: applications
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/company/gitops-apps
targetRevision: main
path: apps/production # ArgoCD finds all apps here
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true # Automatically correct driftManaging 15 Clusters From One ArgoCD Instance
Managing multiple clusters was actually easier than I expected. I registered each cluster with our central ArgoCD instance using argocd cluster add. This command adds the cluster credentials to ArgoCD, and from that point on, I could target any cluster just by specifying destination.server in the application manifest.
The real power came when I started using ApplicationSets. Instead of creating separate application definitions for each cluster, I created templates that automatically generated applications for all registered clusters. When I added a new cluster, ApplicationSet automatically deployed our standard set of infrastructure applications to it. Monitoring, logging, ingress controllers, all of it. This took cluster bootstrapping from a day of manual work to about 10 minutes.
Secrets Management (The Part Everyone Gets Wrong)
Here’s the uncomfortable truth about GitOps: you can’t put plain-text secrets in Git. Obviously. But you also can’t do GitOps properly if your secrets aren’t in Git. This paradox trips up a lot of people.
I chose Sealed Secrets after trying a few other options. The workflow is simple: developers create a regular Kubernetes Secret, encrypt it with kubeseal, and commit the encrypted version to Git. The Sealed Secrets controller running in the cluster is the only thing with the private key needed to decrypt it. So secrets live in Git safely, and ArgoCD can manage them like any other resource.
The first time I had to rotate secrets across all 15 clusters, I appreciated this setup. Update the secret, re-encrypt it, commit to Git, and ArgoCD rolls it out everywhere. No manual kubectl commands across 15 different clusters.
# My process for sealing a secret
kubectl create secret generic mysecret --dry-run=client -o yaml | \
kubeseal -o yaml > sealed-secret.yaml
# Now I can safely commit sealed-secret.yaml to GitProgressive Delivery with Argo Rollouts
For our most critical applications (payment processing, authentication), I added Argo Rollouts for canary deployments. This was after we had a bad deployment that took down payments for 20 minutes. Never again.
With Rollouts, I could define deployment steps: send 10% of traffic to the new version, pause for 5 minutes while monitoring error rates, then 25%, pause, then 50%, then 100%. If error rates spiked at any step, the deployment automatically rolled back. This transformed scary deployments into boring, routine operations.
What Changed
The transformation took about three months to fully roll out, but the results were dramatic.
Deployment time dropped from over 30 minutes of manual kubectl commands to under 5 minutes, fully automated. Developers just merged their PR and ArgoCD handled the rest. We went from deploying once a week (because deployments were painful) to deploying 10+ times per day.
Error rates in deployments fell from 15% to under 2%. The self-healing feature was the main contributor. When someone inevitably made a manual change to production during an incident, ArgoCD would detect the drift and revert it within 3 minutes. This prevented so many “undocumented production changes” incidents.
Rollback time became trivial. Before GitOps, a rollback involved finding the old Helm chart, making sure you had the right version, running helm upgrade, and hoping you didn’t make it worse. Now it’s git revert, wait 2 minutes, done. I’ve done rollbacks in under 60 seconds.
Audit trail was the unexpected benefit. During security audits, I could show our auditors the exact Git history of every infrastructure change. Who made it, when, what the change was, who approved the PR. The auditors loved it.
What I Learned
The app-of-apps pattern is non-negotiable at scale. Without it, you’re manually managing hundreds of Application resources. With it, the system is self-managing.
Secrets management needs to be solved on day one. I’ve seen teams try to “add it later” and they never do. They end up with secrets scattered across different systems, none of them in Git, total chaos. Pick a solution (Sealed Secrets, External Secrets, SOPS, whatever) and commit to it from the start.
Use ArgoCD Projects for RBAC. This lets you give teams access only to their applications without giving them access to everything. We have 12 different teams, each with their own ArgoCD Project. They can see and manage their apps, but they can’t accidentally delete someone else’s.
Start simple. I didn’t try to migrate 200 applications on day one. I picked one low-risk application, proved GitOps worked, showed the team how much easier it made deployments, then used that momentum to expand. Six months later, everything was in ArgoCD.