Argo Rollouts Canary Deployments With Automatic Rollback

Shipping a broken build is not the interesting part. The interesting part is how many requests it serves before something notices and pulls it back.

Original content from computingforgeeks.com - post 170484

An Argo Rollouts canary answers that question. It replaces your Deployment with a controller that shifts traffic in steps and queries Prometheus between each step to decide whether to keep going. This guide builds the whole thing on Amazon EKS with ALB weighted target groups, wires a Prometheus success-rate gate, then measures the blast radius of a deliberately broken release two ways: once behind an Argo Rollouts canary, and once through a plain Kubernetes Deployment with nothing watching. The same release, the same load, the same load balancer. Both paths are delivered through Argo CD, first as Kustomize manifests and then as a Helm chart, because the choice between them changes more than you would expect.

Ran this end to end on a throwaway EKS cluster in August 2026, then deleted it. Every number below came off that cluster.

All manifests, the Helm chart, the demo app, the k6 scripts, and the measurement harness live in c4geeks/argo-rollouts-lab.

1. What a Rollout changes

A Rollout is a drop-in replacement for a Deployment. Same pod template, same selector, same replica count. What changes is the update strategy: instead of RollingUpdate swapping pods as fast as the surge budget allows, you get an ordered list of steps the controller walks through, and it stops walking the moment a step fails.

The practical difference is that a Deployment has no opinion about whether the new version is any good. It replaces pods, reports success, and moves on. A Rollout can pause at 20% traffic, ask Prometheus what the error rate looks like, and abort back to the previous ReplicaSet without anyone being paged. That is the entire value proposition, and section 8 puts a number on it.

Traffic splitting needs a router that understands weights. Argo Rollouts supports ALB, NGINX, Istio, Traefik and several others, with Gateway API available as a plugin. This build uses ALB weighted target groups because on EKS that is what most teams already run, and because it exercises the real integration path: Pod Identity to the load balancer controller to a weighted listener rule. If you are on a service mesh instead, the Istio traffic-shifting setup covers the equivalent wiring.

2. Lab topology and prerequisites

The cluster is deliberately small but not tiny. Argo CD, Argo Rollouts, kube-prometheus-stack, the demo app and an in-cluster k6 job do not coexist on two 4 GB nodes without evictions.

• EKS with 3 worker nodes, 2 vCPU and 8 GB each (t3.large). Two smaller nodes will thrash
• Tested on Kubernetes 1.36.2, Argo Rollouts v1.9.1, Argo CD v3.5.0, AWS Load Balancer Controller v3.5.0, kube-prometheus-stack chart 88.1.5
• The EKS Pod Identity agent addon on the cluster. An OIDC provider is only needed if you use IRSA instead
• Public subnets tagged kubernetes.io/role/elb=1 so the controller can find them
• On your workstation: git, aws, kubectl, helm, jq, envsubst and python3 (the load-test targets use the last two), plus standalone kustomize and docker buildx only if you rebuild the demo images

Sizing drivers for a real deployment are the Prometheus retention window and the scrape interval, not the Rollout controller, which is tiny. This lab scrapes every 5 seconds and keeps 6 hours, which fits comfortably in 1.5 GB. Production clusters retaining weeks of history need storage sized against cardinality, and that is a separate exercise.

Reader-specific values repeat throughout, so set them once:

export CLUSTER="cfg-lab-eks"
export REGION="eu-west-1"
export ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export IMAGE="ghcr.io/c4geeks/rollouts-demo"

# Pinned so the IAM policy and the plugin match the controller you install.
export LBC_VERSION="v3.5.0"       # https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases
export ROLLOUTS_VERSION="v1.9.1"  # https://github.com/argoproj/argo-rollouts/releases

The demo app is purpose-built for this. It renders its own version as a coloured tile so a traffic split is visible in a browser, exposes demo_http_requests_total with a version label for Prometheus, and takes an ERROR_RATE build argument. That last part matters: v3 is a genuinely broken image rather than a config toggle, so “ship the bad build” is reproducible.

The three tags are published and pullable, so the only setup is the repository itself:

git clone https://github.com/c4geeks/argo-rollouts-lab.git
cd argo-rollouts-lab

To point the lab at your own registry instead, reset IMAGE to it, run make -C app push IMAGE="${IMAGE}" to build all three, then repoint the two Kustomize overlays and the chart. The subshells matter, because kustomize edit acts on the current directory and the rest of this guide runs from the repository root.

export IMAGE="your-registry.example.com/rollouts-demo"

(cd manifests/base    && kustomize edit set image ghcr.io/c4geeks/rollouts-demo="${IMAGE}")
(cd manifests/control && kustomize edit set image ghcr.io/c4geeks/rollouts-demo="${IMAGE}")

# BSD sed (macOS) needs an explicit empty extension: sed -i '' "s|...|...|"
sed -i "s|repository: ghcr.io/c4geeks/rollouts-demo|repository: ${IMAGE}|" \
  charts/rollouts-demo/values.yaml

Section 9 hands delivery to Argo CD, which renders from Git rather than from your working copy, so that path needs those edits committed to a fork with the Applications’ repoURL pointed at it. A private registry also needs a docker-registry secret named ghcr in the demo, demo-control and demo-helm namespaces, which is what the imagePullSecrets entry in each manifest refers to. On the public path that secret does not exist, so every demo pod logs a FailedToRetrieveImagePullSecret warning event and then pulls anonymously anyway. It is noise, not a failure.

Two commands below need the Rollouts kubectl plugin, which is a workstation binary rather than a cluster component:

OS=$(uname | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')

curl -fsSL -o kubectl-argo-rollouts \
  "https://github.com/argoproj/argo-rollouts/releases/download/${ROLLOUTS_VERSION}/kubectl-argo-rollouts-${OS}-${ARCH}"
chmod +x kubectl-argo-rollouts
sudo mv kubectl-argo-rollouts /usr/local/bin/

Then bring up the platform. The script is idempotent and guards every IAM create, so it is safe to run at any point, including after working through section 3 by hand:

./platform/install.sh    # IAM, LB controller, Rollouts, Argo CD, kube-prometheus-stack
make deploy-control      # the plain-Deployment control group used in section 8
make dashboards          # the Grafana dashboard behind the screenshots

Section 3 is the by-hand version of the two installs that matter, and it is worth reading even if the script already ran. The one order to avoid is section 3 after the script, because its unguarded aws iam create-policy will error on a policy that already exists.

3. Install Argo Rollouts and the ALB traffic router

If Argo CD is not on the cluster yet, the Argo CD on EKS walkthrough covers bootstrapping it. For a non-EKS cluster the generic Argo CD install applies.

The load balancer controller needs IAM. Pod Identity is the current mechanism and it is less fiddly than the older federation route, though IRSA still works if that is what your cluster is standardised on. Save the trust policy once, because two roles share it:

vim /tmp/pod-identity-trust.json

The principal is the EKS pods service, not an OIDC federation:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}

The controller’s permission set is published with the release, so pull the matching version rather than hand-writing it. Create the role, attach the policy, then map it to the service account:

curl -fsSL -o /tmp/lbc-policy.json \
  "https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/${LBC_VERSION}/docs/install/iam_policy.json"

aws iam create-policy --policy-name CFGLabAWSLoadBalancerControllerPolicy \
  --policy-document file:///tmp/lbc-policy.json

aws iam create-role --role-name CFGLabLBCRole \
  --assume-role-policy-document file:///tmp/pod-identity-trust.json

aws iam attach-role-policy --role-name CFGLabLBCRole \
  --policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/CFGLabAWSLoadBalancerControllerPolicy"

aws eks create-pod-identity-association --cluster-name "${CLUSTER}" \
  --region "${REGION}" --namespace kube-system \
  --service-account aws-load-balancer-controller \
  --role-arn "arn:aws:iam::${ACCOUNT_ID}:role/CFGLabLBCRole"

Skip the attach step and the controller starts cleanly, then fails every load balancer call with AccessDenied and no Ingress ever gets an address.

Now the controller itself. The vpcId flag is not optional here, and the reason is section 10’s first entry:

# chart releases: https://github.com/aws/eks-charts/releases
helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster --name "${CLUSTER}" --region "${REGION}" \
  --query 'cluster.resourcesVpcConfig.vpcId' --output text)

helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --version 3.5.0 \
  --set clusterName="${CLUSTER}" \
  --set region="${REGION}" \
  --set vpcId="${VPC_ID}" \
  --set serviceAccount.create=true \
  --set serviceAccount.name=aws-load-balancer-controller \
  --wait

Argo Rollouts needs AWS credentials of its own if you want target group verification, which makes the controller confirm a weight change actually landed in AWS before advancing to the next step. Get this wrong and it degrades quietly: the controller emits a TargetGroupVerifyError event carrying the AWS error text, then a WeightVerifyError, and carries on with the canary unverified. Check for both rather than assuming a quiet run means it worked. Grant the permissions first:

vim /tmp/rollouts-elb-policy.json

Read-only ELB describes are all it needs:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "elasticloadbalancing:DescribeTargetGroups",
      "elasticloadbalancing:DescribeTargetHealth",
      "elasticloadbalancing:DescribeLoadBalancers",
      "elasticloadbalancing:DescribeListeners",
      "elasticloadbalancing:DescribeRules",
      "elasticloadbalancing:DescribeTags"
    ],
    "Resource": "*"
  }]
}

Same pattern as before, pointed at the argo-rollouts service account:

aws iam create-policy --policy-name CFGLabRolloutsELBReadPolicy \
  --policy-document file:///tmp/rollouts-elb-policy.json

aws iam create-role --role-name CFGLabRolloutsRole \
  --assume-role-policy-document file:///tmp/pod-identity-trust.json

aws iam attach-role-policy --role-name CFGLabRolloutsRole \
  --policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/CFGLabRolloutsELBReadPolicy"

aws eks create-pod-identity-association --cluster-name "${CLUSTER}" \
  --region "${REGION}" --namespace argo-rollouts \
  --service-account argo-rollouts \
  --role-arn "arn:aws:iam::${ACCOUNT_ID}:role/CFGLabRolloutsRole"

The region is as load-bearing as the role. Argo Rollouts builds its ELB client from the default credential chain, Pod Identity injects credentials but no region, and this cluster’s pods cannot read IMDS, so AWS_REGION has to be set explicitly on the controller:

# chart releases: https://github.com/argoproj/argo-helm/releases
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

helm upgrade --install argo-rollouts argo/argo-rollouts \
  --namespace argo-rollouts --create-namespace \
  --version 2.41.1 \
  --set dashboard.enabled=true \
  --set controller.awsVerifyTargetGroup=true \
  --set 'controller.extraEnv[0].name=AWS_REGION' \
  --set "controller.extraEnv[0].value=${REGION}" \
  --wait

Confirm the plugin is on your path. It reports the client build only, so check the controller’s image separately:

kubectl argo rollouts version

kubectl -n argo-rollouts get deploy argo-rollouts \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Both the plugin build and its commit hash print:

kubectl-argo-rollouts: v1.9.1+b6bd3bc
  BuildDate: 2026-07-17T09:37:17Z
  GitCommit: b6bd3bcf8f60d717a98763d26acc983db7f97cb0
  GitTreeState: clean
  GoVersion: go1.24.13
  Compiler: gc
  Platform: darwin/arm64

quay.io/argoproj/argo-rollouts:v1.9.1

If the plugin and the controller drift apart the plugin usually still works, but the dashboard can render fields the older controller never populates. Keep them on the same minor release.

4. The Rollout, three Services, and the use-annotation Ingress

The listings below are abridged to the fields that carry the argument. The full files, including probes, pull secrets and the latency gate, are in the repository. ALB traffic splitting needs three Services and one Ingress with an unusual backend. The three Services all select the same app label, but Argo Rollouts pins the stable and canary ones to specific ReplicaSet hashes during a rollout, which is what gives each version its own target group.

sudo vim manifests/base/services.yaml

Three near-identical Services, distinguished by a role label the ServiceMonitors key off later:

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-root
  namespace: demo
  labels:
    app: rollouts-demo
    role: root
spec:
  selector:
    app: rollouts-demo
  ports:
    - name: http
      port: 80
      targetPort: http

Repeat that block for rollouts-demo-stable and rollouts-demo-canary, changing only the name and the role label. The Ingress is where the ALB contract lives:

sudo vim manifests/base/ingress.yaml

The port name use-annotation tells the controller to ignore the backend and read a forward action out of an annotation instead. Argo Rollouts owns that annotation and rewrites the weights on every step:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rollouts-demo
  namespace: demo
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/group.name: cfg-lab
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "5"
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "2"
spec:
  ingressClassName: alb
  rules:
    - host: demo.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: rollouts-demo-root
                port:
                  name: use-annotation

Both healthcheck values are set on purpose. Leave the timeout out and the target group is rejected, which section 10 covers.

The Rollout itself looks like a Deployment until strategy:

sudo vim manifests/base/rollout.yaml

Four weight steps, each followed by an analysis gate that has to pass before the next one runs:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
        - name: demo
          image: ghcr.io/c4geeks/rollouts-demo:v1
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet: { path: /healthz, port: http }
            initialDelaySeconds: 2
            periodSeconds: 3
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              memory: 128Mi
  strategy:
    canary:
      canaryService: rollouts-demo-canary
      stableService: rollouts-demo-stable
      trafficRouting:
        alb:
          ingress: rollouts-demo
          rootService: rollouts-demo-root
          servicePort: 80
      steps:
        - setWeight: 20
        - analysis: &gate
            templates:
              - templateName: success-rate
            args:
              - name: canary-service
                value: rollouts-demo-canary
        - setWeight: 40
        - analysis: *gate
        - setWeight: 60
        - analysis: *gate
        - setWeight: 80
        - analysis: *gate

Apply it and the controller creates two target groups, one per Service, and binds them to the listener rule:

kubectl apply -k manifests/base
kubectl get targetgroupbindings -n demo \
  -o custom-columns=NAME:.metadata.name,SERVICE:.spec.serviceRef.name

Two bindings, one stable and one canary, is the sign the wiring is correct:

NAME                               SERVICE
k8s-demo-rollouts-17e1c8a9f9       rollouts-demo-canary
k8s-demo-rollouts-f73941d198       rollouts-demo-stable

If only one binding appears, the Rollout has not reconciled the canary Service yet. If none appear, the Ingress never produced a load balancer, and the events on it will say why.

The gate in the next section asks Prometheus about the canary pods alone, and it can only do that if the canary Service is scraped under its own name. Two ServiceMonitors, keyed on the role label, give the operator what it needs:

sudo vim manifests/base/servicemonitors.yaml

The Prometheus Operator adds a service label from the selected Service, which is the label the analysis query filters on:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rollouts-demo-canary
  namespace: demo
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: rollouts-demo
      role: canary
  endpoints:
    - port: http
      path: /metrics
      interval: 5s

Duplicate that for role: stable. Both are needed, and section 5 explains the one way that bites you.

5. The analysis gate that does not roll back healthy releases

This is the section worth reading twice, because the obvious query is wrong in a way that only shows up under real traffic.

Nearly every Argo Rollouts example computes success rate as good requests over total requests:

sum(rate(demo_http_requests_total{service="rollouts-demo-canary",code!~"5.."}[30s]))
/
sum(rate(demo_http_requests_total{service="rollouts-demo-canary"}[30s]))

There are two empty states and they behave differently. A canary pod that has never served a request exports no counter series at all, so the query returns an empty result. A canary that served traffic and then went quiet still has the series, but the rate over the window is zero on both sides of the division, and Prometheus returns NaN. NaN >= 0.95 is false, so with failureLimit: 0 the controller reads it as a failed measurement and rolls back a release that was never broken. Ask Prometheus directly during that quiet window:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 &
PF_PID=$!
sleep 3

curl -s --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(demo_http_requests_total{service="rollouts-demo-canary",code!~"5.."}[30s])) / sum(rate(demo_http_requests_total{service="rollouts-demo-canary"}[30s]))' \
  | jq -r '.data.result[0].value[1]'

Not a number, which is what the gate then compares against 0.95:

NaN

The fix is to measure the error ratio instead and clamp the denominator, so an idle canary reads as healthy rather than as undefined:

sudo vim manifests/analysis/success-rate.yaml

len(result) == 0 covers the earlier window where the pods exist but have not exported a counter yet, and clamp_min covers the window where they have exported one but no traffic has arrived:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo
spec:
  args:
    - name: canary-service
  metrics:
    - name: success-rate
      interval: 20s
      count: 3
      initialDelay: 30s
      failureLimit: 0
      successCondition: len(result) == 0 || result[0] >= 0.95
      provider:
        prometheus:
          address: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
          query: |
            1 - (
              (sum(rate(demo_http_requests_total{service="{{args.canary-service}}",code=~"5.."}[30s])) or vector(0))
              /
              clamp_min(sum(rate(demo_http_requests_total{service="{{args.canary-service}}"}[30s])), 1)
            )

The {{args.canary-service}} placeholder is Argo Rollouts templating, not PromQL, so substitute the real Service name to check it by hand:

curl -s --get http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=1 - ((sum(rate(demo_http_requests_total{service="rollouts-demo-canary",code=~"5.."}[30s])) or vector(0)) / clamp_min(sum(rate(demo_http_requests_total{service="rollouts-demo-canary"}[30s])), 1))' \
  | jq -r '.data.result[0].value[1]'

kill "${PF_PID}"

The idle canary now reads as healthy instead of undefined:

1

The clamp makes the gate slightly optimistic when the canary is receiving under one request per second, which is the correct trade: an idle canary is not evidence of a bad release. At the 20% step of a 50 rps test the canary sees roughly 10 rps, so the clamp never engages during real measurement.

One more Prometheus trap, and it is easy to miss because it does not break anything, it just lies. The stable and canary Services both select every pod when no rollout is in progress, so two ServiceMonitors scrape each pod twice. A verified 50 rps load test renders as 100 rps on a dashboard. Deduplicate per pod before summing:

sum by (version) (
  max by (version, pod, path, code) (rate(demo_http_requests_total{namespace="demo"}[30s]))
)

The analysis gate itself is unaffected because it queries the canary Service alone, and during a rollout the two selectors are disjoint. Only whole-application dashboards double count.

6. Watch an Argo Rollouts canary under sustained load

Two Make targets drive this: make load starts the in-cluster k6 job against the ALB, and make e1 runs the whole experiment, meaning it establishes the v1 baseline, starts the load, promotes to v2, records the weight timings and collects the k6 summary. If you would rather drive it by hand, kubectl argo rollouts set image rollouts-demo -n demo "demo=${IMAGE}:v2" is the promotion.

With 50 requests per second running through the ALB, promoting v1 to v2 walks all four steps. The tile grid the demo app renders makes the split visible: each tile is one response, coloured by whichever version answered.

The Rollouts dashboard shows the same thing from the controller’s side, including which analysis runs have passed:

The run completed in 337 seconds with all four gates successful. k6 counted 36,001 responses: 10,779 from the old version, 25,191 from the new one, and 31 failures.

Those 31 failures are worth dwelling on, because they contradict the usual claim that a canary is free. Every one of them was an HTTP 503 with no application error behind it, produced as the ALB deregistered targets while ReplicaSets scaled. A successful, fully gated canary still cost 0.086% of traffic. Small, but not zero, and worth knowing before you promise an SLO.

The harness also timestamped the canary weight in three places: what the controller decided, what it wrote into the Ingress annotation, and what was actually live on the ALB listener rule. Across three weight changes the annotation reached the load balancer in 1.0, 1.3 and 2.1 seconds. That is faster than the tens of seconds ALB weight changes are often assumed to take, though the sampler polls about once per second, so treat roughly one second as the measurement floor rather than a precise figure. The practical consequence is that step pause durations do not need padding for ALB propagation.

7. Ship a deliberately broken release

The v3 image returns HTTP 500 on 10% of requests. Promoting it with the same load running is the entire point of the exercise.

kubectl argo rollouts set image rollouts-demo -n demo "demo=${IMAGE}:v3"
kubectl argo rollouts get rollout rollouts-demo -n demo --watch

It never reached the 40% weight step. The first analysis run failed, the controller aborted, and traffic went back to the previous ReplicaSet:

The controller’s own event log records the decision, which is what you want in a postmortem:

Warning  AnalysisRunFailed  Step Analysis Run 'rollouts-demo-7844fcf9c6-3-1'
                            Status New: 'Failed' Previous: 'Running'
Warning  RolloutAborted     Rollout aborted update to revision 3: Step-based analysis
                            phase error/failed: Metric "success-rate" assessed Failed
                            due to failed (1) > failureLimit (0)

Start to finish, detection and rollback took 39 seconds. Over the whole run k6 saw 18,000 responses, of which the broken version served 233. Splitting the failures by status code matters here: 19 were real application 500s from the bad build, and 26 were 503s from the load balancer shuffling targets during the rollback. Reporting the raw total of 45 would have overstated the damage the release itself did by more than double.

8. The same broken release with no canary

A number is only meaningful next to a control. The control group runs the identical app and the identical v3 image as a plain Deployment with RollingUpdate, behind the same ALB, under the same 50 rps k6 profile. Nothing watches it.

kubectl set image deployment/control-demo -n demo-control "demo=${IMAGE}:v3"
kubectl rollout status deployment/control-demo -n demo-control

It reported success in 13 seconds, which is exactly the problem. Fast, clean, and now serving errors to everyone:

Side by side, on the same cluster, the same day:

Measurement	Argo Rollouts canary	Plain Deployment
Total requests	18,000	17,901
Requests served by the broken version	233 (1.29%)	15,800 (88.26%)
Application 500s	19	1,547
Load balancer 5xx during the change	26	207
Overall failure rate	0.25%	9.798%
Outcome	Aborted in 39s, previous version kept	Completed in 13s, stayed broken

The broken build produced 81 times more application errors without progressive delivery than with it, and served 68 times more responses from the bad version. Counting every failure including the load balancer’s own 5xx, the gap is still 39 times. Neither run involved a human. That ratio is the argument to take to whoever asks why a rollout controller is worth the operational surface.

Grafana shows the shape of both events. The canary ramp climbs cleanly, the abort appears as a single narrow error spike that ends the moment the analysis fails, and the control group’s error rate sits on a plateau until the run ends:

Worth noting what the canary panel does not show: a long flat stretch of degraded service. The spike is narrow because the gate fired on the first measurement it had enough data to judge.

9. The same Rollout as a Helm chart through Argo CD

Everything so far was plain Kustomize. The second delivery path templates the Rollout as a Helm chart, and the interesting part is not the templating, it is that the canary shape becomes data.

Instead of a fixed list of steps in YAML structure, the chart takes weights and gates as values, so a dev cluster can ship a single 50% step while production walks four:

canary:
  steps:
    - weight: 20
      analysis: true
    - weight: 40
      analysis: true
    - weight: 60
      analysis: true
    - weight: 80
      analysis: true

The template walks that list and expands each entry:

      steps:
        {{- range .Values.canary.steps }}
        - setWeight: {{ .weight }}
        {{- if and .analysis $.Values.analysis.enabled }}
        - analysis:
            templates:
              - templateName: {{ include "rollouts-demo.fullname" $ }}-success-rate
            args:
              - name: canary-service
                value: {{ include "rollouts-demo.fullname" $ }}-canary
        {{- end }}
        {{- end }}

The gotcha here is the analysis query. Argo Rollouts uses {{args.canary-service}} for its own substitution, and Helm will happily eat those braces before Argo Rollouts ever sees them. Escape them so Helm emits the literal string:

sum(rate(demo_http_requests_total{service="{{ "{{args.canary-service}}" }}"}[30s]))

Rendering the chart and diffing it against the Kustomize output confirms both paths produce the same four weight steps and the same four gates. If you are still on Helm 3, the Helm 4 migration notes cover what changes in templating behaviour.

Handing both paths to Argo CD is where the payoff shows. Apply the two Applications, each pointing at a different path in the same repository:

kubectl apply -f gitops/

kubectl get applications -n argocd \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

Both should settle to Synced and Healthy, deployed to separate namespaces on the same load balancer:

Argo CD ships a built-in health check for the Rollout kind, so an Application stays Progressing for as long as the canary is stepping and only flips to Healthy once the rollout fully promotes. That is what makes a Rollout safe to drive from GitOps: sync does not mean shipped. The mapping is worth knowing in full, because a paused rollout surfaces as Suspended rather than Progressing, and the abort from section 7 shows up as Degraded. The resource tree exposes the AnalysisTemplate and every AnalysisRun as first-class children:

One setting matters more than it looks: leave selfHeal off. With self-healing on, Argo CD reverts the image change that kubectl argo rollouts set image makes and fights the controller mid-canary. Drive promotions by changing the tag in Git, or by patching the Application’s values, not by mutating the live Rollout. For fleets of these across clusters, ApplicationSets generate the Applications rather than templating them by hand, and the Flux and Argo CD comparison covers the equivalent Flux wiring.

10. What broke, and the fixes

Every one of these cost real time on a real cluster.

Error: “failed to fetch VPC ID from instance metadata: context deadline exceeded”

The load balancer controller crashlooped immediately on a stock EKS managed nodegroup:

{"level":"error","logger":"setup","msg":"unable to initialize AWS cloud",
 "error":"failed to get VPC ID: failed to fetch VPC ID from instance metadata:
 error in fetching vpc id through ec2 metadata: get mac metadata:
 operation error ec2imds: GetMetadata, canceled, context deadline exceeded"}

Managed nodegroups built on the default AL2023 AMI without a custom launch template ship with an IMDS hop limit of 1, so a pod sitting one hop out on the pod network cannot reach instance metadata at all. Nodes from a custom launch template usually get 2. Confirm it on your own:

aws ec2 describe-instances --region "${REGION}" \
  --filters Name=tag:eks:cluster-name,Values="${CLUSTER}" \
  --query 'Reservations[].Instances[].MetadataOptions.HttpPutResponseHopLimit' \
  --output text

A row of ones means the controller will never reach IMDS:

1	1	1

Passing --set vpcId= and --set region= together, as section 3 does, removes the IMDS dependency entirely. Raising the hop limit to 2 also works, but it is the larger change and it applies to every pod on the node.

Error: “Health check timeout ‘5’ must be smaller than the interval ‘5’”

The Ingress reconciled but no ALB appeared, and the events repeated this every second:

Warning  FailedDeployModel  Failed deploy model due to operation error Elastic Load
Balancing v2: CreateTargetGroup, api error ValidationError: Health check timeout '5'
must be smaller than the interval '5'

Setting a 5 second healthcheck interval without also lowering the timeout leaves both at 5, and AWS rejects the target group. Set healthcheck-timeout-seconds to something below the interval, as the Ingress in section 4 does.

A healthy release rolls back on its very first gate

Covered in section 5, and it is the failure most likely to reach production quietly, because it only fires when the canary has not received traffic yet. Symptoms are an abort on the first analysis with a success-rate metric that never printed a number. Rewrite the query as an error ratio with a clamped denominator.

Dashboards report exactly double the real request rate

A load test verified at 50 rps rendering as 100 rps on a Grafana panel is not a load generator problem. Two ServiceMonitors over Services whose selectors overlap outside a rollout scrape every pod twice. Wrap the rate in max by (version, pod, path, code) before summing, which has the useful property of fixing already-recorded data rather than only new scrapes. Keep every real dimension in that inner grouping. Collapsing down to just the pod also merges the status codes, so the result is the pod’s single busiest series rather than its request rate, and the error ratio computed from it comes out wrong.

Two more that cost less time but are worth pre-empting. kubectl cp cannot pull a results file out of a Job whose pod has already completed, so write summaries to stdout and recover them from the logs. And when the ALB action annotation and the live listener rule are polled sequentially in one loop, the measured propagation lag comes out jittered and occasionally negative, because a single describe-rules call takes about a second; poll each source on its own thread with its own timestamp.

If you want to reproduce any of this, the repository has a Makefile that stands up the cluster, installs the platform, and runs each experiment as a single target. Tear it down when you are done, because an idle cluster with a NAT gateway and an ALB still bills.