The art of a boring deployment

A deployment is easier to operate when its failure conditions are defined before traffic moves. For a Kubernetes service, that means checking compatibility, reserving rollout capacity, and deciding which signals stop promotion. Rollback needs the same preparation.

Start with the failure boundary #

A deployment is a change in a running system, not just a new container image. Its actual boundary includes database compatibility, queue semantics, configuration, and the expectations of every consumer.

Before changing anything, write down what the previous version still needs to understand. If a schema migration removes a column that the previous release reads, a healthy set of pods does not mean you can safely roll back.

Give the rollout a contract #

“Looks healthy” is too vague to automate. Define a short list of conditions that should remain true during the rollout. Each needs an owner, a measurement window, and an explicit response.

These are different questions. A readiness probe should be cheap and local. It should not cascade every downstream outage into a total loss of ready instances.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-api
spec:
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  minReadySeconds: 20
  progressDeadlineSeconds: 300

These strategy settings keep the desired replica count available and require each new pod to remain ready for 20 seconds. maxSurge: 1 also requires enough spare capacity for an additional pod. progressDeadlineSeconds reports a stalled rollout; Kubernetes does not automatically roll it back. Application correctness and schema compatibility need separate checks.

Observe the transition, not just the destination #

The steady state gets most of our diagrams. The transition gets most of our incidents. During a rollout, old and new versions coexist, caches are cold, and traffic distribution can be uneven.

Attach the release identifier to logs, metrics, and traces. Compare cohorts over a window long enough to contain meaningful traffic. For a low-volume service, this may take longer than a fixed five-minute gate.

For an availability target $S$ over a window of $T$ minutes, the time-based budget is:

A 99.9% target over 30 days gives 43.2 minutes under a time-based definition. A request-based objective has a different denominator; do not interchange the two.

Design the way back #

A rollback is another deployment. It deserves the same compatibility checks and observability as the forward change. Keep the previous artifact addressable by immutable digest, and record the configuration that accompanied it.

def may_promote(sample_count, error_rate, baseline):
    if sample_count < 1_000:
        return False  # Insufficient evidence is not success.
    return error_rate <= baseline + 0.002

The gate requires 1,000 observations and permits an absolute error-rate increase of 0.2 percentage points. That tolerance must come from the service’s error budget. At low baseline error rates, 1,000 requests may provide too little evidence; use confidence intervals and account for correlated failures before automating promotion.

Record the operational contract #

Record the artifact, the owner, the observable success conditions, and the recovery procedure in the same place. Prefer a short runbook exercised during a calm afternoon to a comprehensive document discovered during an outage.

A release record should let the on-call engineer identify the change, assess its impact, and choose a recovery action without reconstructing the deployment from logs.¹