Progressive Delivery in Production: Argo Rollouts, Flagger, and Canary Deployments That Actually Protect You

The deployment that convinced me to never again trust readiness probes as my only safety net happened in year twelve of my career. We were pulling a monolith apart into services, and someone merged a change that accidentally capped the database connection pool size at 10 instead of 100. The service passed all its readiness checks because it could still connect to the database. It passed all its health checks. The Kubernetes rolling update proceeded normally, replacing pods one by one. By the time the deployment finished, 100% of traffic was hitting a service that was quietly timing out 8% of database-bound requests.

We had SLO burn happening in real time and no automated mechanism had stopped it. We caught it twelve minutes later from a Grafana alert. By then, the damage was done.

That incident is why I now route every critical service deployment through a progressive delivery controller. Not because canary deployments are simple to operate (they aren’t, at first) but because they’re the only mechanism that asks the one question that actually matters: is this new version behaving correctly under real production traffic, not just in tests?

The Gap That Standard Kubernetes Rollouts Leave Open

When you trigger a RollingUpdate, Kubernetes replaces pods from the old ReplicaSet with pods from the new one, checking only that each replacement pod passes its readiness probe before moving on. A readiness probe is binary: can this pod accept traffic or not? It tells you almost nothing about whether the code inside is actually working for your users.

A pod serving 500 errors can be “ready.” A pod returning the wrong data can be “ready.” A pod with a slow query that degrades under load will pass readiness checks at low traffic and fall apart at full load, and the rollout will proceed because all pods are technically healthy.

The maxUnavailable and maxSurge settings give you some control over rollout speed but do nothing to answer the fundamental question: what percentage of real user requests are succeeding on the new version, and how does that compare to the baseline?

That is the question progressive delivery tools are built to answer.

What Progressive Delivery Actually Means

People use “canary deployment” loosely to mean many things, including “deploy to one server first.” What I mean by progressive delivery is specifically traffic-based canary analysis: you route a configurable percentage of actual production traffic to the new version, measure how that traffic is behaving using real observability data, and only promote the rollout if the metrics look good.

This is fundamentally different from pod-based strategies. In a pod-based canary (which is what vanilla Kubernetes does), if you have 10 pods and update 1, roughly 10% of requests hit the canary. But the traffic split is unstable: as Kubernetes replaces more pods, the ratio shifts automatically. There is no way to hold the canary at 10% for 30 minutes while you analyze metrics, then move to 25%, then to 50%.

True traffic-based progressive delivery requires either a service mesh (Istio, Linkerd, Cilium) or an ingress controller with traffic weighting support (NGINX, Traefik, or the Kubernetes Gateway API). The progressive delivery controller sits between your CI/CD system and these traffic management layers, orchestrating the percentage shifts and automating the analysis.

The two main tools in this space are Argo Rollouts and Flagger. I have used both in production. They solve the same problem from different architectural angles.

Argo Rollouts: Explicit Control at Every Step

Argo Rollouts replaces the standard Kubernetes Deployment resource with a Rollout custom resource. This is the key design choice that separates it from Flagger: instead of watching an existing Deployment, Argo Rollouts is itself the deployment controller.

A basic Rollout manifest for a canary strategy:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  replicas: 10
  strategy:
    canary:
      trafficRouting:
        istio:
          virtualService:
            name: payment-service-vsvc
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: success-rate-check
        - setWeight: 30
        - pause: {duration: 10m}
        - setWeight: 100

The setWeight steps adjust the traffic percentage via the Istio VirtualService (or your chosen routing mechanism). The pause steps hold the rollout at that weight for a defined duration. The analysis step triggers an AnalysisRun that queries metrics and returns pass or fail.

The AnalysisTemplate is where the real power lives:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{
              job="payment-service",
              status!~"5..",
              version="{{args.canary-hash}}"
            }[5m])) /
            sum(rate(http_requests_total{
              job="payment-service",
              version="{{args.canary-hash}}"
            }[5m]))            

This template queries Prometheus every minute and requires a 99% success rate on the canary version. If it fails once, the AnalysisRun fails, and Argo Rollouts automatically rolls back by sending all traffic back to the stable ReplicaSet and scaling down the canary.

What I appreciate about this approach is explicitness. Each step is declared, the order is enforced, and the controller makes no decisions beyond the analysis pass/fail logic. You can pause a rollout manually, promote it immediately with kubectl argo rollouts promote, or abort it with kubectl argo rollouts abort. That level of control matters during real incidents when you need to act without waiting for automation.

Argo Rollouts integrates natively with ArgoCD through the rollout health check mechanism, which means your GitOps sync loop in ArgoCD correctly shows a rollout as “Progressing” while it steps through canary analysis, and marks it “Healthy” only when the final promotion step completes. If you are already running ArgoCD, Argo Rollouts is the natural choice.

Flagger: Controller-Loop Progressive Delivery

Flagger takes a fundamentally different approach. You keep your existing Kubernetes Deployment, and Flagger watches it. When you change the Deployment (by updating the image tag), Flagger intercepts the change, creates canary infrastructure alongside the stable version, and manages the traffic shift through its own analysis loop.

The Flagger Canary resource declares your analysis configuration:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  service:
    port: 80
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 30s

When Flagger detects a new image in the Deployment, it creates a checkout-service-canary Deployment (a copy at the new image version) and starts shifting traffic toward it in 10% increments every minute. It measures both success rate and P99 latency on the canary. If either metric exceeds the failure threshold more than 5 times, Flagger rolls back and marks the canary as failed, blocking further automated promotion until a human intervenes.

Flagger works with Istio, Linkerd, AWS App Mesh, Contour, Nginx, Traefik, and the Kubernetes Gateway API. The traffic routing integration is pluggable, giving it broader applicability than Argo Rollouts in environments that don’t run a full service mesh.

The main thing to understand about Flagger’s design is that you don’t explicitly declare rollout steps. The controller decides when to promote based on the analysis results, advancing the canary weight automatically at each analysis interval. This is more hands-off than Argo Rollouts. For teams that want automated deployment to be truly autonomous, this is an advantage. For teams that want to manually promote between steps or pause at specific weights, Argo Rollouts gives more control.

Choosing Between Them

I have run both tools across different organization types. My recommendation:

Use Argo Rollouts if you are already in the ArgoCD ecosystem (the native integration is seamless), you want explicit auditable step-by-step control over rollout progression, or your team has compliance requirements where every deployment state needs to be traceable.

Use Flagger if you are using Flux for GitOps (Flagger was built alongside Flux by the Weaveworks team), you want progressive delivery to be fully automated without manual promotion gates, or you are running Linkerd (Flagger’s Linkerd integration is particularly clean).

In practice, both tools solve the problem well. The ecosystem fit matters more than the feature comparison. I have never regretted choosing either one when the context matched.

Building Analysis Templates That Actually Protect You

Here is where most teams get this wrong: they set up Argo Rollouts or Flagger, point it at an HTTP success rate metric, and feel like they are safe. HTTP success rate at 99% sounds rigorous, but it misses a large class of real problems.

The failure mode I see most often is business-logic errors that don’t surface as HTTP 5xx responses. A payment service returning the wrong confirmation status returns HTTP 200. A recommendation engine serving stale embeddings returns HTTP 200. A checkout flow that silently discards cart items returns HTTP 200. Your success rate analysis will pass all of them.

The fix is adding business metrics to your analysis templates. This requires your application to emit custom metrics, which is more work upfront but pays off every time it catches something that success rate would miss.

For a payment service, I want to see:

• payment_transactions_total with status="completed" vs status="failed" (not just HTTP codes)
• cart_abandonment_rate if you can measure it from events
• checkout_session_duration_p99 to catch latency regressions in the business flow

These metrics need to come from application instrumentation, not just the ingress layer. If you’re running the Prometheus, Loki, and Grafana observability stack, adding application-level counters and histograms is straightforward with any Prometheus client library.

The other mistake is setting analysis intervals too short. I have watched teams configure 30-second analysis intervals for services that handle mostly batch requests, resulting in high metric variance that triggers false rollbacks. For most production services, I use:

• 2 to 5 minute analysis interval
• 5 to 10 minute pause at each traffic weight step
• Failure limit of 2 to 3 (allow some metric noise before failing the rollout)
• Total canary duration of at least 20 minutes before full promotion

This means a full rollout takes roughly 30 to 45 minutes from start to 100% traffic. That is the cost of safety. If you need faster deployments, use feature flags for the high-risk code paths and reserve progressive delivery for service-level image updates. The feature flags and progressive delivery guide covers when each approach is appropriate.

Integration with GitOps and Your Observability Pipeline

In a GitOps-driven setup, progressive delivery changes your deployment loop in a meaningful way. When you update an image tag in Git and ArgoCD syncs the change, you need to know not just that the sync succeeded but that the rollout analysis passed.

Argo Rollouts exposes rollout status as part of the ArgoCD application health check, which means your CI pipeline can run argocd app wait my-app --health and it will wait until the rollout fully completes or fails. This makes progressive delivery transparent to your existing CI tooling without requiring any changes to how you trigger deployments.

For notifications, both Argo Rollouts and Flagger support webhook-based alerting. I always configure Slack notifications for rollout started (with the image tag), step promotion, analysis failure (with the failing metric value), and rollback completion. Receiving a message that says “payment-service rolled back at 10% traffic: request-duration P99 exceeded 300ms (measured: 847ms)” gives your team immediate context without anyone having to dig through dashboards.

For the analysis metrics provider, Prometheus is the most common choice and works well with both tools. Datadog is supported by Argo Rollouts via its metrics provider interface, which is useful in organizations that have standardized on Datadog rather than self-hosted Prometheus.

The Traffic Routing Layer

Progressive delivery only works if your traffic routing layer supports weighted splits. This is worth understanding before you start, because the right choice depends on what you are already running.

If you are using Istio, the VirtualService resource handles traffic weighting natively, and both Argo Rollouts and Flagger have first-class Istio integrations. This is my recommended setup for teams already running a service mesh. The service mesh deep dive explains how VirtualService traffic management works under the hood.

If you are using Cilium without a service mesh, Argo Rollouts supports Cilium’s Gateway API implementation for traffic splitting. This is a strong option for teams that moved away from sidecars toward eBPF-native networking. The Cilium in production guide covers the relevant networking architecture.

If you are using NGINX Ingress, both tools support canary annotations on the Ingress resource for traffic weighting. This works but has real limitations: NGINX Ingress canary weighting is approximate, works by sending a percentage of requests to a secondary Ingress backend rather than precise weight-based routing, and is harder to combine with header-based routing for A/B testing.

If you are adopting the Kubernetes Gateway API, this is now the recommended path for new setups. The HTTPRoute resource handles traffic weighting with proper weight-based splitting, which is architecturally cleaner than NGINX annotations and gives you consistent behavior across different gateway implementations.

What the Connection Pool Incident Actually Taught Me

Back to that database connection pool problem. After we implemented Argo Rollouts with proper analysis templates, we replayed the scenario in a staging environment with production-like load. The connection pool exhaustion drove P99 latency from our baseline of 120ms to over 800ms within two minutes of sustained traffic.

The analysis template had a max: 300ms threshold on request duration with a one-minute interval. The rollout failed at the 10% traffic step, sent all traffic back to the stable version, and sent a Slack notification: “payment-service rollback: request-duration P99 exceeded 300ms (measured: 847ms).”

What made the analysis template effective wasn’t HTTP success rate. The service was still returning HTTP 200 for most requests because it was timing out database calls and returning cached partial data, which is a realistic production failure mode. It was the latency metric that caught it.

In twenty years of running production systems, I have found that latency is the most reliable early indicator of a bad deployment. Error rates lag behind latency almost always. By the time your error rate climbs, you have often already burned significant SLO budget. Latency starts moving within the first few minutes of traffic hitting a slow dependency.

Build your analysis templates around latency first. Add error rate second. Add business metrics third. In that order of priority. And make sure your SLO thresholds in Prometheus align with what you’re protecting in your error budget framework, so that a rollback also has a corresponding impact on burn rate calculations.

Getting Started Without Boiling the Ocean

The biggest mistake teams make when adopting progressive delivery is trying to implement it everywhere at once. I have watched teams spend weeks configuring Argo Rollouts across 40 services only to get burned by configuration drift and analysis templates that were too sensitive for some services and too lenient for others.

My recommendation: start with the two or three services that have caused the most production incidents from bad deployments. Configure progressive delivery for those services, tune the analysis templates over three or four real rollouts until the sensitivity feels right, then use those templates as the baseline for your other services.

The services where a bad deployment would burn your error budget fastest are where progressive delivery provides the most value. Start there. It’s a 2-to-3 week investment to get the first service right, including proper analysis templates, Slack notifications, and GitOps integration. After that, adding additional services takes a fraction of the time.

For the underlying infrastructure, you need: a service mesh or Gateway API implementation for traffic splitting, Prometheus for metrics, Argo Rollouts or Flagger deployed in your cluster, and applications emitting meaningful metrics beyond HTTP status codes.

The operational overhead is real. You are adding custom controllers, CRDs, and a dependency on your metrics infrastructure to the deployment path. If Prometheus is down when you try to deploy, your analysis templates won’t have data. Build fallback behavior using inconclusiveLimit in Argo Rollouts, which will pause the rollout and wait for human decision rather than auto-failing on missing data.

Progressive delivery won’t prevent every bad deployment. It will prevent the ones where your service starts behaving badly under production traffic in measurable ways, which is most of them. After running it in production across multiple organizations, I would not ship critical services any other way.