Canary and Blue/Green Deployments Without the Foot-Guns

The deployment was healthy. Kubernetes said so. Every pod was ready, the rollout completed, and the dashboards were mostly green. Then support reported that customers were seeing errors.

The new version had passed its readiness probe because it could answer /health. It could not reliably process the larger requests produced by one particular customer workflow. The canary received too little of that traffic to move the aggregate error rate, promotion happened automatically, and the faulty build reached every pod.

Nothing in that sequence was unusual. That is the problem.

Progressive delivery is often presented as a traffic-routing feature. Send 5 percent to the new version, wait, then increase it. In production, the difficult part is not moving traffic. It is deciding whether the new version deserves more traffic, preserving the evidence behind that decision, and stopping safely when reality disagrees with the plan.

A rollout is a control system, not a deployment style

Blue/green and canary releases solve different problems.

Blue/green gives you two complete environments or application revisions. One serves production traffic while the other waits for validation. The switch is usually fast, which makes the technique useful when partial traffic would create inconsistent behaviour or when testing needs a complete production-shaped stack.

A canary exposes the new revision gradually. That limits the initial blast radius and gives telemetry time to reveal regressions. It works well when requests can be divided safely and the signal from a small traffic sample is meaningful.

Neither technique automatically makes a release safe. Both need a closed feedback loop:

change -> deploy -> expose -> observe -> decide -> promote or abort

Most foot-guns live in the last three steps. Teams automate deployment and exposure, then leave observation and promotion as vague operational rituals.

Blue/green is only reversible while both sides still work

A common blue/green implementation switches a Kubernetes Service selector from one version to another:

kubectl patch service checkout \
 --type merge \
 -p '{"spec":{"selector":{"app":"checkout","track":"green"}}}'

That is easy to understand and easy to regret. The command changes live state outside the normal ci/cd or GitOps path. The repository still describes blue, the cluster serves green, and the next reconciliation may switch traffic back without anyone intending it.

Put the routing decision in the deployment controller. Argo Rollouts, Flagger, service meshes, and cloud load balancers can all manage it, but there must be one owner. If Argo Rollouts controls promotion, a pipeline job should not also patch the Service, and an operator should not need a handwritten command during every release.

Blue/green rollback also becomes fiction once the new version writes data that the old version cannot read. Keep the previous revision running until the observation window closes, but treat schema compatibility as the real rollback boundary. Pods are disposable. Data is not.

A canary needs enough traffic to tell you something

This Argo Rollouts fragment is mechanically valid:

strategy:
 canary:
 steps:
 - setWeight: 5
 - pause: {duration: 10m}
 - setWeight: 25
 - pause: {duration: 15m}
 - setWeight: 50
 - pause: {duration: 20m}

It says nothing about safety. A ten-minute pause might contain thousands of representative requests, or three health checks and an engineer opening the homepage.

Promotion rules should account for request volume as well as elapsed time. Low-traffic services may need synthetic transactions, request mirroring, or a longer observation period. High-traffic services may need segmentation because an overall success rate can hide a failure limited to one endpoint, region, tenant class, or payload shape.

Do not judge a canary with one broad metric. At minimum, examine request failures, latency, saturation, restarts, and a business-level success signal such as completed checkouts or accepted jobs. Compare the canary with the stable revision over the same interval. A fixed threshold like “error rate below 1 percent” can approve a regression when both versions are already unhealthy.

Readiness probes are not release analysis

A readiness probe answers a narrow question: should this pod receive traffic now? It does not prove that the release is correct.

We keep seeing probes that check only whether an HTTP process responds. Better probes verify the local dependencies required to serve traffic, but they should still stay cheap and predictable. Turning readiness into a full integration test creates a different failure mode, where a slow downstream system removes every pod from service at once.

Release analysis belongs outside the pod lifecycle. For example, Argo Rollouts can query Prometheus before continuing:

metrics:
 - name: canary-error-ratio
 interval: 1m
 successCondition: result[0] < 0.01
 failureLimit: 2
 provider:
 prometheus:
 address: http://prometheus.monitoring:9090
 query: |
 sum(rate(http_requests_total{version="canary",code=~"5.."}[5m]))
 /
 sum(rate(http_requests_total{version="canary"}[5m]))

Production configuration needs protection against an empty denominator, missing series, delayed ingestion, and low sample counts. “No data” must not silently mean “good”. Test the analysis query before trusting it with promotion authority.

GitOps still needs an emergency brake

GitOps gives the release an auditable desired state, but reconciliation can fight incident response. If an operator aborts a rollout while the repository continues requesting the bad version, the controller may try again.

Define the emergency procedure before the incident. Decide whether aborting modifies the rollout object, reverts the Git commit, suspends reconciliation, or performs some combination. Record who can do it and how the repository is brought back into agreement afterward.

The break-glass path should be short enough to use under pressure:

kubectl argo rollouts abort checkout
kubectl argo rollouts status checkout --watch

Those commands should be rehearsed in a non-production environment. If they exist only in a runbook nobody has opened, they are documentation, not a safety mechanism.

Database changes decide whether rollback is real

Use expand and contract migrations. Add the new schema first, deploy code that can work with old and new representations, migrate data separately, then remove the old schema only after the previous application version is no longer a rollback target.

Avoid coupling destructive migrations to application startup. Five new pods racing to alter the same table is not progressive delivery. Run migrations as an explicit, observable pipeline stage with locking, timeouts, and a recorded result.

The same warning applies to message formats, caches, and background workers. During a canary, old and new consumers often run together. Version events, make handlers tolerant of unknown fields, and ensure jobs are not processed twice merely because both colours are active.

Measure delivery outcomes, not just rollout mechanics

DORA metrics can show whether the delivery system is improving: deployment frequency, lead time for changes, change failure rate, and recovery performance expose different parts of the system. They are not canary promotion signals. A release controller should decide from current service behaviour, while DORA metrics reveal whether the wider devops process is producing smaller changes, fewer failures, and faster recovery.

This is where platform engineering earns its keep. Teams should receive a paved rollout pattern with hardened CI/CD, GitOps ownership, infrastructure as code, observability, and tested recovery controls. The same principles apply across AWS, Azure, GCP, OCI, and on-prem Kubernetes, even when the traffic-routing implementation differs.

If you only do one thing this week

Take one production service and write down its promotion contract. Name the metrics, minimum request volume, observation window, abort thresholds, data compatibility requirement, and person or controller authorised to promote.

Then run a deliberately bad release through a staging environment. Make it return errors on one endpoint, confirm that analysis fails, abort it, and verify that Git and the cluster agree afterward.

If that exercise feels uncomfortable, good. It has found uncertainty during office hours instead of during an incident bridge.

If this is on your plate, TecLeads does exactly this as part of our DevOps & Platform Engineering work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.