GitOps and CI/CD: From 45-Minute Builds to Continuous Deployment

Jan 20, 2025

I’ve spent the last four years transforming our deployment process from manual kubectl commands and 45-minute build pipelines to a fully automated GitOps system with sub-15-minute deployments. This wasn’t a weekend hackathon project. It was a gradual evolution driven by pain, production incidents, and the realization that manual deployments don’t scale.

Let me tell you how I got here. Three years ago, deployments worked like this: a developer would finish a feature, push to the main branch, wait 45 minutes for CI to build and test, then manually run kubectl commands to deploy to production. Sometimes they’d forget a step. Sometimes they’d apply the wrong manifest. Sometimes they’d deploy to the wrong namespace. Every deployment was a potential incident waiting to happen.

I was tired of being paged at 2 AM because someone deployed the wrong container image. I was tired of deployment delays because the only person who knew the kubectl incantations was on vacation. I decided to fix this systematically.

The Evolution: Three Phases of Automation

This transformation happened in three distinct phases, each solving a specific pain point. I’ll walk you through them in the order I implemented them, because that’s probably the order you’ll need them too.

Phase 1: CI/CD Pipeline Optimization (Making Builds Not Suck)

The Problem: Our CI/CD pipeline took 45 minutes to run. Developers would push code, then go get coffee, check Slack, attend a meeting, and come back to see if the build passed. The feedback loop was brutally slow. When a build failed (and they often did), it would take another 45 minutes to validate the fix.

The Wake-Up Call: I analyzed our pipeline and found that we were doing incredibly wasteful things. We built Docker images from scratch every time, even when nothing changed. We ran the entire test suite sequentially, even though tests were independent. We uploaded artifacts to S3 after every stage, including intermediate artifacts nobody ever used.

The Solution: I optimized the CI/CD pipeline using every trick I knew. Multi-stage Docker builds with layer caching. Parallel test execution across multiple runners. Smarter caching of dependencies. Elimination of unnecessary artifact uploads. The results were dramatic.

The pipeline went from 45 minutes to 12 minutes. The Docker build stage went from 15 minutes to 3 minutes through aggressive layer caching. Test execution went from 20 minutes to 6 minutes through parallelization. Developers stopped context-switching during builds because the feedback loop was fast enough to keep their attention.

Read the full guide: CI/CD Pipeline Optimization

What you’ll learn:

Phase 2: GitOps with ArgoCD (Declarative Deployments)

The Problem: Even with fast builds, deployments were still manual. Someone had to run kubectl commands. Someone had to remember which environment variables to set. Someone had to validate the deployment succeeded. This was error-prone and didn’t scale. We needed deployments to be automatic and declarative.

The Philosophy Shift: I’d been reading about GitOps and the idea resonated immediately. What if the Git repository was the single source of truth for everything running in the cluster? What if deployments were just Git commits? What if rollbacks were just Git reverts? This would eliminate an entire class of human errors.

The Solution: I implemented ArgoCD for GitOps-based deployments. Every Kubernetes manifest lived in a Git repository. ArgoCD watched those repositories and automatically synchronized any changes to the cluster. A developer could deploy to production by merging a pull request. No kubectl commands. No SSH access to production. Just Git.

The first time I watched ArgoCD automatically detect a drift (someone had manually edited a deployment in production) and automatically revert it to match the Git repository, I knew this was the right approach. Self-healing infrastructure.

I set up multiple environments (dev, staging, production) with environment-specific Kustomize overlays. Same base manifests, different configurations per environment. This eliminated configuration drift between environments.

Read the full guide: GitOps with ArgoCD

What you’ll learn:

Phase 3: Advanced GitOps with Flux2 (Enterprise-Grade Automation)

The Problem: ArgoCD worked great, but as our infrastructure grew, we needed more sophisticated capabilities. We had multiple Git repositories. We had dependencies between applications (deploy this only after that is healthy). We had complex promotion workflows (dev to staging requires manual approval, staging to production is automatic).

The Requirements: We needed a GitOps system that could handle multiple Git sources, support complex dependency graphs, integrate with our existing GitLab infrastructure, and provide fine-grained control over deployment workflows.

The Solution: I implemented Flux2 for advanced GitOps workflows. Flux2 is more modular and flexible than ArgoCD. It uses a collection of specialized controllers (source-controller, kustomize-controller, helm-controller) that work together.

I set up Flux2 to monitor multiple Git repositories. Our application manifests lived in application repos. Our infrastructure configs lived in a central ops repo. Flux2 would automatically detect changes in any of these repos and reconcile the cluster state.

For complex deployments, I used Flux dependencies. The backend service wouldn’t deploy until the database migration job completed successfully. The frontend wouldn’t deploy until the backend was healthy. This prevented partial deployments that would cause outages.

For promotions, I set up image update automation. When a new image was pushed to our container registry, Flux2 would automatically update the dev environment manifests, wait for health checks to pass, then create a pull request to promote to staging. This automated our entire deployment pipeline while still giving us manual approval gates where we needed them.

Read the full guide: Flux2 GitOps with GitLab and Kubernetes

What you’ll learn:

The Complete GitOps Toolkit

Here’s how I use these systems in practice today:

For standard services: Flux2 with automated image updates. Developers push code, CI builds and tests, pushes image to registry, Flux2 detects new image, updates manifests, deploys automatically.

For infrastructure changes: GitOps with manual approval. Changes to namespace configs, RBAC policies, or cluster-wide resources require a pull request review before Flux2 deploys them.

For database migrations: Flux dependencies ensure migrations run before application deployments. The application won’t deploy if the migration fails.

For multi-environment promotions: Automated dev deployments, PR-based staging promotions with manual approval, automated production promotions after staging soaks for 24 hours.

Always monitored: ArgoCD/Flux2 health dashboards, deployment notifications in Slack, automated rollback on health check failures.

What This Actually Achieved

The numbers tell the story better than I can:

Deployment metrics:

Developer experience:

More importantly, deployments stopped being a special event. They became routine. A developer can deploy a fix at 4 PM on Friday without stress because the process is fully automated and has rollback built in. The system catches bad deployments automatically.

Lessons Learned the Hard Way

Start with CI/CD optimization before GitOps. Fast pipelines are essential. If your builds take 45 minutes, GitOps won’t save you. You need the foundation of a fast, reliable CI/CD pipeline before you add GitOps on top.

GitOps is a philosophy, not just a tool. It’s the idea that Git is the single source of truth. This means you need to stop making manual changes to production. If someone kubectl applies something directly to the cluster, ArgoCD will revert it. This is painful at first (people will complain), but it’s what makes the system work.

Secrets management is the hard part of GitOps. You can’t commit plain-text secrets to Git. I use sealed secrets (bitnami-labs/sealed-secrets) which encrypt secrets that can only be decrypted by the cluster. This lets us commit encrypted secrets to Git safely.

Environment parity through Kustomize is worth the investment. Having the same base manifests with environment-specific overlays means dev, staging, and production are nearly identical. This catches environment-specific bugs before they reach production.

Progressive delivery is the natural next step after GitOps. Once you have automated deployments through GitOps, you want automated canary releases, automated rollbacks on metrics, A/B testing. This is what Flux2’s Flagger integration provides.

Image update automation changed everything. Before this, we still had to manually update image tags in manifests. Flux2 image automation eliminated that manual step. Now the entire flow from git push to production deployment is automated.

The ROI on Automation

Implementing these systems took about 8 months of part-time work spread across a year. The return on investment was immediate:

Time savings:

Quality improvements:

Business impact:

The initial time investment was significant, but these systems have been running for years with minimal maintenance. The time saved on manual deployments paid back the investment within the first quarter.

Where to Start

If you’re still doing manual deployments or have slow CI/CD pipelines, here’s the path I’d recommend:

  1. Optimize your CI/CD pipeline first (get builds under 15 minutes)
  2. Implement basic GitOps with ArgoCD (learn the GitOps workflow)
  3. Add environment promotion workflows (dev to staging to production)
  4. Graduate to Flux2 for advanced workflows when you need them
  5. Add progressive delivery (canary releases, automated rollbacks)

Don’t try to implement everything at once. Each phase builds on the previous one. Master fast CI/CD before attempting GitOps. Get comfortable with basic GitOps before adding complex promotion workflows.

Read the guides in the order listed above. Each one assumes you understand the previous concepts. The guides include real code, real configurations, and real troubleshooting advice based on problems I actually encountered.

The Stack Today

Here’s what our fully automated deployment system looks like today:

CI/CD: GitLab CI with multi-stage Docker builds, parallel testing, and artifact caching. 12-minute average build time.

GitOps: Flux2 managing 100+ applications across dev, staging, and production. Automated image updates with manual promotion gates where needed.

Progressive Delivery: Flagger for automated canary releases with Prometheus metric analysis and automatic rollback.

Secrets: Sealed Secrets for GitOps-compatible secret management.

Monitoring: Prometheus for metrics, ArgoCD health dashboard, Slack notifications for all deployments.

This stack handles 40+ deployments per day across all environments with minimal human intervention. It’s not perfect, but it’s been running reliably for two years, and it’s transformed how we ship software.

If you want to dive deeper into specific aspects of our deployment automation:

This is how you build confidence in your deployment process. Not by being perfect, but by having systems that catch mistakes automatically and enable rapid iteration.

✦ El Muhammad's Portfolio

© 2025 Aria

Instagram YouTube TikTok 𝕏 GitHub