TecLeads TecLeads Blog
2026-11-13 · 7 min read

GitOps with Argo CD That Survives Production

Rows of server racks lit in blue in a data center
devopsgitopsargo-cdkubernetesplatform-engineering

At 02:13, the production deployment was green in CI, red in Argo CD, and half the pods were still running the previous image. Someone had patched a Deployment during an earlier incident, the Git repository disagreed, and automated sync was trying to put everything back while the team was debugging.

That is GitOps in name only. Git exists, Argo CD exists, and nobody can say which system owns production.

Good GitOps in 2026 is not a repository full of YAML with an Argo logo attached. It is an operating model with strict ownership, boring promotion, bounded automation, and enough telemetry to explain what happened without opening five browser tabs.

Git must be allowed to win

The first rule is blunt: if Git declares a field, people and other controllers do not get to mutate that field behind its back.

This is where many installations go wrong. A CI/CD pipeline runs kubectl set image, an operator changes a replica count, an image updater writes directly to the cluster, and an engineer applies a hotfix from a laptop. Argo CD then reports drift, but nobody knows whether reconciliation is safe.

Decide ownership at field level. Horizontal Pod Autoscaler owns live replica counts. Argo CD owns the HPA configuration. An operator may own generated resources, while Git owns the custom resource that caused them to exist. These boundaries should be reflected in manifests and documented ignore rules, not handled through tribal memory.

Before enabling automated sync, inspect the proposed change:

argocd app diff payments-prod
argocd app sync payments-prod --dry-run
kubectl diff -k environments/prod/payments

Ignore differences narrowly. Ignoring an entire Deployment because one controller changes one annotation is how genuine drift gets buried.

Emergency changes still happen. The production rule should be: patch, stabilise, then immediately capture or revert the patch in Git. A cluster edit without a follow-up commit is borrowed time.

CI builds artifacts, Argo deploys them

A clean division between CI and GitOps removes a surprising amount of confusion.

CI tests the code, scans dependencies and images, builds an immutable artifact, signs it, and updates the desired version through a pull request. Argo CD reconciles the approved state into Kubernetes. CI should not also deploy the same workload with helm upgrade or kubectl apply.

Promote image digests, not floating tags:

image:
 repository: registry.example.com/payments
 digest: sha256:8c6b0f1b2e6f5a7c...

A digest answers the question that matters during an incident: what exact artifact is running? Tags such as main, stable, and even reused semantic versions do not.

We generally prefer one mainline configuration repository with explicit environment directories over a branch per environment. Branches hide the real difference between staging and production. Directories make it visible in one review:

apps/payments/base/
environments/dev/payments/
environments/staging/payments/
environments/prod/payments/

Promotion becomes a small pull request that changes the digest and any intentional environment configuration. The diff is the release record. Required review, policy checks, manifest rendering, schema validation, and signature verification belong on that path.

The platform team owns the paved road

A platform engineering team should not hand every product team an empty repository and call it self-service. That merely distributes YAML archaeology.

The platform should provide the ApplicationSet patterns, repository structure, policy checks, observability hooks, and rollback procedure. Product teams should own application configuration within those boundaries. Ownership should be clear enough that an incident commander knows whom to call without consulting the Git history.

Argo CD projects are one of the most useful boundaries and one of the most frequently skipped:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
 name: payments
 namespace: argocd
spec:
 sourceRepos:
 - https://git.example.com/platform/payments-config.git
 destinations:
 - namespace: payments-*
 server: https://kubernetes.default.svc
 clusterResourceWhitelist:
 - group: ""
 kind: Namespace
 namespaceResourceBlacklist:
 - group: rbac.authorization.k8s.io
 kind: ClusterRoleBinding

The exact policy will differ, but the default should deny access outside the application's source repositories, clusters, namespaces, and resource types. An Application should not become an indirect cluster administrator because its repository was trusted once.

Use separate Argo CD control planes when the failure domains or trust boundaries justify it. A single instance controlling every development and production cluster may be convenient, but it also creates one credential concentration and one reconciliation failure domain. At minimum, separate production from lower environments and protect the management plane like production infrastructure.

Automated sync needs brakes

Automated sync is useful. Unbounded automated sync is an incident generator with excellent consistency.

Use sync waves for dependencies, health checks that represent application readiness, and progressive delivery where a failed rollout must stop before reaching every replica. Argo Rollouts, service mesh traffic controls, or a cloud load balancer can handle traffic movement. Argo CD should remain the source reconciliation engine, not a home-grown deployment script disguised as hooks.

Pruning deserves particular care. Enable it after repository ownership and deletion behaviour are understood. A renamed directory, broken generator, or empty rendered chart can otherwise become a very efficient deletion request. Protect namespaces, persistent volumes, and shared resources explicitly. Test destructive changes in a representative non-production cluster.

Sync windows are useful for protecting staffed periods, but they are not a substitute for safe releases. If the system can deploy only while its most experienced engineer is watching, the delivery design is unfinished.

Secrets do not belong in readable Git

GitOps does not mean putting plaintext credentials into Git. Nor does base64 make a Secret safe.

Use an external secret manager such as AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp Vault, or an on-prem equivalent. External Secrets Operator and the Secrets Store CSI Driver are common ways to connect Kubernetes workloads to those systems. If encrypted secrets must live in Git, use a tool such as SOPS with tightly controlled keys and a documented recovery process.

Argo CD repository credentials, cluster credentials, signing keys, and administrative access deserve the same treatment. Disable casual local admin use, connect single sign-on, apply least privilege, and retain audit logs outside the Argo CD pod lifecycle.

Measure delivery, not controller activity

Argo CD exposes useful operational signals: application health, sync status, reconciliation duration, queue depth, and controller errors. Alert on sustained failures and reconciliation lag. A brief out-of-sync state during a normal change is not automatically an incident.

Do not mistake sync counts for delivery performance. DORA metrics need events with business meaning. Deployment frequency should count successful production releases, not reconciliation retries. Lead time should connect the code change to the production artifact and deployment. Change failure rate needs incident or rollback data. Recovery time needs a defined service restoration event.

Join CI metadata, Git commits, artifact digests, Argo CD application revisions, and incident records. Without that chain, the dashboard may be attractive while answering the wrong question.

The same model should work across AWS, Azure, GCP, OCI, and on-prem Kubernetes. Cloud-specific identity and networking will differ, but artifact promotion, policy, reconciliation, and evidence should not be reinvented for every cluster.

If you only do one thing this week

Pick one production Application and write down who owns every mutation path. Check CI jobs, operators, autoscalers, image automation, Helm hooks, and human access. Remove any second writer for fields that Git is supposed to control.

That exercise usually finds the real GitOps problem faster than another tool installation. Argo CD works best when it has authority, boundaries, and evidence. Without those, it is just another participant in the argument over what production should look like.


If this is on your plate, TecLeads does exactly this as part of our DevOps & Platform Engineering work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.

📍 Tech Pulse · today's quick question 🟡 Level: Intermediate Cybersecurity

What is a zero-day?

Pick an answer to see how other engineers voted.

Want a hand with this?

TecLeads helps engineering teams ship faster and more securely.

Book a 30-minute call

← All posts