Designing Canary and Progressive-Delivery Gates for Plan Rollouts
A canary gate replaces the hard binary block with a bounded, self-correcting rollout: it routes a small slice of live traffic onto a candidate plan, watches real p95 latency and error rate against the anchored baseline, and then promotes, holds, or rolls back automatically. This guide specifies the verdict-plus-telemetry contract that drives that decision, a full asyncio controller that steps traffic and evaluates each window, and the exact step schedule, observation windows, and rollback triggers that keep the rollout honest without a human in the loop.
Progressive delivery exists because a scored regression verdict is a prediction, not a measurement. The threshold evaluator can classify a plan as safe from optimizer cost alone, but only production traffic reveals cache behaviour, lock contention, and parameter skew. This stage sits at the boundary between the two: it consumes the scored verdict and live query telemetry, and it emits promote, hold, or rollback decisions to the deploy controller. It never re-scores plans and never mutates baselines — it decides how much traffic a candidate has earned.
Architectural Boundaries
Upstream, the gate consumes two independent streams. The first is the scored verdict emitted by Tuning Thresholds for False Positive Reduction: a PASS or WARN here is a precondition, not a promotion — a FAIL verdict never reaches the canary at all, because there is no point routing live traffic onto a plan the offline evaluator has already rejected. The second stream is live telemetry: candidate p95 latency and error rate scraped from a Prometheus/OpenTelemetry query, keyed by the same deterministic plan fingerprint used everywhere else in the pipeline.
Downstream, the gate emits a single decision to the deploy controller — an Argo Rollouts AnalysisRun, a GitHub Actions job status, or an internal traffic-shaping service. The decision is one of three verbs: promote the candidate to the next traffic step, hold at the current step for another observation window, or roll back to the anchored baseline. The controller owns none of the traffic-shaping mechanics itself; it owns only the arithmetic that turns a window of live metrics into one of those three verbs. That separation is what makes the gate testable — you can replay a recorded telemetry window and get a deterministic decision, exactly as the offline regression rule engine is replayable.
The failure-isolation boundary is strict: if either input stream is unavailable — the verdict store times out or the metric backend returns no samples — the gate must emit hold, never promote. A rollout that cannot see its own signal is a rollout that must not advance.
Deterministic Routing and Schema Enforcement
Every step decision is a pure function of one typed input: the scored verdict joined to the observed telemetry window. The controller validates that joined payload against a schema before it will act, because a missing sample_count silently treated as zero is the difference between “hold on cold metrics” and “promote on nothing”.
The observation payload is validated against this JSON Schema before the decision runs:
{
"$id": "https://queryplan.org/schemas/canary-observation.json",
"type": "object",
"additionalProperties": false,
"required": ["query_hash", "verdict", "step_index", "baseline_p95_ms", "candidate_p95_ms", "candidate_error_rate", "sample_count"],
"properties": {
"query_hash": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
"verdict": { "type": "string", "enum": ["PASS", "WARN"] },
"step_index": { "type": "integer", "minimum": 0, "maximum": 3 },
"baseline_p95_ms": { "type": "number", "exclusiveMinimum": 0 },
"candidate_p95_ms": { "type": "number", "minimum": 0 },
"candidate_error_rate": { "type": "number", "minimum": 0, "maximum": 1 },
"sample_count": { "type": "integer", "minimum": 0 }
}
}Traffic is split by a stable, coordination-free routing formula. A request lands on the candidate plan when a hash of its sticky key falls below the current step percentage, so the same session sees a consistent plan for the life of the step:
route_to_candidate = (zlib.crc32(session_key.encode()) % 100) < step_schedule[step_index]
The step ladder itself is a fixed list, step_schedule = [5, 25, 50, 100], and the promotion transition is next_index = min(step_index + 1, len(step_schedule) - 1). Because both the routing hash and the step transition are pure integer arithmetic, a replayed telemetry window always yields the same routing set and the same decision — the property that lets you unit-test a rollout offline. The decision function itself is equally small: rollback dominates promote, and hold dominates both when the sample is cold, so no window can ever both roll back and promote.
Production-Ready Implementation
The controller below is an asyncio service. It reads the candidate’s live p95 and error rate from the Prometheus HTTP API, reads the anchored baseline p95 through an asyncpg pool, steps traffic through the schedule, and emits a promote/hold/rollback decision per window. Every window is wrapped in an OpenTelemetry span, metrics are exported as counters and a gauge, and all logs are structured through structlog. The evaluation is pure; the only I/O is the two read paths and the traffic-shaping call.
from __future__ import annotations
import asyncio
import os
import zlib
from dataclasses import dataclass
from enum import Enum
from typing import Literal
import asyncpg
import httpx
import structlog
from opentelemetry import metrics, trace
log = structlog.get_logger(__name__)
tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
DECISION_TOTAL = meter.create_counter("canary_decision_total", unit="1")
ROLLBACK_TOTAL = meter.create_counter("canary_rollback_total", unit="1")
STEP_GAUGE = meter.create_up_down_counter("canary_traffic_step_pct", unit="%")
# --- Fixed rollout policy (mirrors canary.yaml) ---------------------------
STEP_SCHEDULE: tuple[int, ...] = (5, 25, 50, 100)
OBSERVATION_WINDOW_S = 600 # 10 minutes per step
P95_BUDGET_RATIO = 1.2 # candidate p95 must stay <= 1.2x baseline
ERROR_RATE_CEIL = 0.005 # 0.5% error-rate ceiling
MIN_SAMPLE_COUNT = 500 # guard against cold-metric promotion
PROM_TIMEOUT_S = 4.0
class Decision(str, Enum):
PROMOTE = "promote"
HOLD = "hold"
ROLLBACK = "rollback"
@dataclass(frozen=True)
class Observation:
query_hash: str
verdict: Literal["PASS", "WARN"]
step_index: int
baseline_p95_ms: float
candidate_p95_ms: float
candidate_error_rate: float
sample_count: int
def decide(obs: Observation) -> Decision:
"""Pure decision: rollback dominates, cold metrics hold, else promote."""
breached_latency = obs.candidate_p95_ms > obs.baseline_p95_ms * P95_BUDGET_RATIO
breached_errors = obs.candidate_error_rate >= ERROR_RATE_CEIL
if breached_latency or breached_errors:
return Decision.ROLLBACK
if obs.sample_count < MIN_SAMPLE_COUNT:
return Decision.HOLD # never promote on cold metrics
return Decision.PROMOTE
def route_to_candidate(session_key: str, step_index: int) -> bool:
return (zlib.crc32(session_key.encode()) % 100) < STEP_SCHEDULE[step_index]
class CanaryController:
def __init__(self, pool: asyncpg.Pool, http: httpx.AsyncClient, prom_url: str) -> None:
self._pool = pool
self._http = http
self._prom = prom_url.rstrip("/")
async def _prom_scalar(self, promql: str) -> float | None:
resp = await self._http.get(
f"{self._prom}/api/v1/query",
params={"query": promql},
timeout=PROM_TIMEOUT_S,
)
resp.raise_for_status()
result = resp.json()["data"]["result"]
return float(result[0]["value"][1]) if result else None
async def _baseline_p95(self, query_hash: str) -> float:
row = await self._pool.fetchrow(
"SELECT baseline_p95_ms FROM plan_baseline WHERE query_hash = $1",
query_hash,
)
if row is None:
raise LookupError(f"no anchored baseline for {query_hash}")
return float(row["baseline_p95_ms"])
async def _observe(self, query_hash: str, step_index: int) -> Observation | None:
p95_q = f'histogram_quantile(0.95, sum(rate(query_latency_ms_bucket{{fp="{query_hash}",track="candidate"}}[5m])) by (le))'
err_q = f'sum(rate(query_errors_total{{fp="{query_hash}",track="candidate"}}[5m])) / sum(rate(query_total{{fp="{query_hash}",track="candidate"}}[5m]))'
cnt_q = f'sum(increase(query_total{{fp="{query_hash}",track="candidate"}}[10m]))'
baseline, p95, err, cnt = await asyncio.gather(
self._baseline_p95(query_hash),
self._prom_scalar(p95_q),
self._prom_scalar(err_q),
self._prom_scalar(cnt_q),
)
if p95 is None or cnt is None:
return None # metric store returned nothing -> caller holds
return Observation(
query_hash=query_hash,
verdict="WARN",
step_index=step_index,
baseline_p95_ms=baseline,
candidate_p95_ms=p95,
candidate_error_rate=err or 0.0,
sample_count=int(cnt),
)
async def _shape_traffic(self, query_hash: str, pct: int) -> None:
# Idempotent set-weight call into the deploy controller.
await self._pool.execute(
"""
INSERT INTO canary_state (query_hash, traffic_pct, updated_at)
VALUES ($1, $2, now())
ON CONFLICT (query_hash)
DO UPDATE SET traffic_pct = EXCLUDED.traffic_pct, updated_at = now()
""",
query_hash,
pct,
)
async def run(self, query_hash: str) -> Decision:
step_index = 0
while step_index < len(STEP_SCHEDULE):
pct = STEP_SCHEDULE[step_index]
with tracer.start_as_current_span("canary.window") as span:
span.set_attribute("query_hash", query_hash)
span.set_attribute("step_pct", pct)
await self._shape_traffic(query_hash, pct)
STEP_GAUGE.add(pct, {"query_hash": query_hash})
await asyncio.sleep(OBSERVATION_WINDOW_S)
try:
obs = await self._observe(query_hash, step_index)
except (LookupError, httpx.HTTPError, asyncpg.PostgresError) as exc:
span.record_exception(exc)
log.warning("canary_observe_failed", query_hash=query_hash, error=str(exc))
DECISION_TOTAL.add(1, {"decision": "hold", "reason": "signal_unavailable"})
continue # fail safe: hold this step, retry the window
if obs is None:
DECISION_TOTAL.add(1, {"decision": "hold", "reason": "no_samples"})
continue
decision = decide(obs)
DECISION_TOTAL.add(1, {"decision": decision.value})
span.set_attribute("decision", decision.value)
log.info(
"canary_decision",
query_hash=query_hash,
step_pct=pct,
decision=decision.value,
candidate_p95_ms=round(obs.candidate_p95_ms, 2),
baseline_p95_ms=round(obs.baseline_p95_ms, 2),
error_rate=round(obs.candidate_error_rate, 5),
samples=obs.sample_count,
)
if decision is Decision.ROLLBACK:
ROLLBACK_TOTAL.add(1, {"query_hash": query_hash})
await self._shape_traffic(query_hash, 0)
return Decision.ROLLBACK
if decision is Decision.PROMOTE:
step_index += 1
# HOLD falls through and re-observes the same step.
return Decision.PROMOTE
async def build(dsn: str, prom_url: str) -> CanaryController:
pool = await asyncpg.create_pool(dsn=dsn, min_size=2, max_size=8, command_timeout=5.0)
http = httpx.AsyncClient()
return CanaryController(pool, http, prom_url)Two properties keep this safe under real traffic. First, any failure to read a signal — a Prometheus timeout, a missing baseline row, an empty result set — resolves to hold, so the rollout stalls in place instead of advancing blind. Second, the candidate is only ever promoted when the window carries at least MIN_SAMPLE_COUNT (500) requests, which is what stops a low-traffic 5% step from promoting on three lucky requests.
Threshold Reference
These are the numbers the rollout is wired against. The step schedule, the per-step observation window, and the promotion criteria are policy, not suggestions; a change to any of them changes the risk profile of every rollout that uses this gate.
| Parameter | Value | Meaning |
|---|---|---|
| Step schedule | Traffic fraction routed to the candidate per step. | |
| Observation window | 10 min / step | Minimum dwell time before a decision is taken. |
| Promotion p95 | ≤ 1.2× baseline | Candidate p95 ceiling to advance a step. |
| Promotion error rate | < 0.5% | Candidate error-rate ceiling to advance a step. |
| Minimum samples | ≥ 500 / window | Below this the window holds, never promotes. |
| Rollback trigger | p95 > 1.2× baseline or error rate ≥ 0.5% | Immediate revert to the anchored baseline. |
| Full-rollout dwell | 2 clean windows at 100% | Windows at 100% before the candidate is anchored. |
The rollback trigger is deliberately a single-window, OR-combined condition: a canary that has already earned live traffic is more dangerous than one that has not, so the gate errs toward reverting fast. Wire the automatic revert as a Prometheus alert so it fires even if the controller process itself is wedged:
groups:
- name: canary-gate
rules:
- alert: CanaryLatencyRegression
expr: |
(
histogram_quantile(0.95, sum(rate(query_latency_ms_bucket{track="candidate"}[5m])) by (le, fp))
/
on(fp) group_left
max(plan_baseline_p95_ms) by (fp)
) > 1.2
for: 3m
labels:
severity: page
action: rollback
annotations:
summary: "Canary p95 above 1.2x baseline for fp {{ $labels.fp }} — auto-rollback"
runbook: "Controller should have reverted; if traffic_pct > 0, force weight to 0 manually."
- alert: CanaryErrorBudgetBurn
expr: |
sum(rate(query_errors_total{track="candidate"}[5m])) by (fp)
/
sum(rate(query_total{track="candidate"}[5m])) by (fp) >= 0.005
for: 2m
labels:
severity: page
action: rollback
annotations:
summary: "Candidate error rate >= 0.5% for fp {{ $labels.fp }} — auto-rollback"Failure Scenarios and Root Cause Analysis
1. Premature promotion on cold metrics. Symptom: a 5% step promotes to 25% within one window even though the candidate served almost no traffic. Root cause: the observation window closed before enough requests accrued, so sample_count was tiny and unrepresentative. Diagnose the per-window sample count:
curl -sG "$PROM_URL/api/v1/query" \
--data-urlencode 'query=sum(increase(query_total{track="candidate"}[10m]))' \
| jq -r '.data.result[].value[1]'Mitigation: the MIN_SAMPLE_COUNT guard of 500 already forces a hold below that floor; for genuinely low-traffic queries, extend OBSERVATION_WINDOW_S rather than lowering the sample floor, so the decision still rests on real load.
2. Metric window too short to see the regression. Symptom: the candidate promotes cleanly, then p95 blows out at 100% under load that the earlier steps never generated. Root cause: a 5-minute rate() range on a spiky workload smooths away the tail the canary was supposed to catch. Diagnose by comparing the windowed p95 against an unsmoothed max:
SELECT query_hash, baseline_p95_ms, last_candidate_p95_ms, updated_at
FROM canary_state
WHERE traffic_pct = 100 AND last_candidate_p95_ms > baseline_p95_ms
ORDER BY updated_at DESC;Mitigation: keep the PromQL rate() range at or above the scrape interval times four, and require two clean windows at 100% before anchoring — one clean window is not proof.
3. Rollback storm / flapping. Symptom: the same candidate rolls back, gets re-submitted, promotes, and rolls back again on a loop. Root cause: a candidate hovering exactly at baseline crosses the threshold on noise, and nothing debounces the re-submission. Diagnose with the rollback counter:
curl -sG "$PROM_URL/api/v1/query" \
--data-urlencode 'query=increase(canary_rollback_total[1h])' \
| jq -r '.data.result[] | "\(.metric.query_hash) \(.value[1])"'Mitigation: add a cooldown before re-canarying a fingerprint that rolled back within the last hour, and treat a fingerprint with three rollbacks in an hour as a hard FAIL back to the offline evaluator — the tuning stage described in Tuning Thresholds for False Positive Reduction owns that escalation, not the canary loop.
4. Sticky-session skew hiding a regression. Symptom: candidate p95 looks healthy at every step but aggregate p95 across the fleet degrades. Root cause: the CRC-based routing consistently sends the same low-cost sessions to the candidate, so the 5% slice is not representative of the parameter distribution. Diagnose by comparing the candidate’s row-estimate distribution against the baseline track’s. Mitigation: salt the routing key per rollout so the candidate cohort is re-drawn each time, and verify the candidate slice’s parameter quartiles match the baseline slice before trusting the p95.
5. Verdict or metric-store timeout resolves to hold. Symptom: a rollout sits at one step indefinitely with repeated canary_observe_failed logs. Root cause: the asyncpg baseline read or the Prometheus query is timing out, and the controller is correctly failing safe to hold. Diagnose the pool and the backend:
SELECT state, count(*) FROM pg_stat_activity
WHERE application_name = 'canary_controller'
GROUP BY state;Mitigation: this is the gate working as designed — the fix is to restore the signal path (raise the pool ceiling, add a read replica for plan_baseline, check Prometheus availability), never to change the default from hold to promote.
Configuration Reference
| Key / Env Var | Default | Purpose |
|---|---|---|
step_schedule | [5, 25, 50, 100] | Ordered traffic percentages per canary step. |
observation_window_s | 600 | Seconds a step dwells before a decision. |
p95_budget_ratio | 1.2 | Candidate p95 ceiling as a multiple of baseline. |
error_rate_ceil | 0.005 | Candidate error-rate ceiling to promote. |
min_sample_count | 500 | Requests required in a window to allow a promote. |
prom_timeout_s | 4.0 | Timeout on each Prometheus query before failing to hold. |
full_rollout_dwell | 2 | Clean windows at 100% before anchoring the candidate. |
CANARY_DSN | — | Connection string for the baseline and canary-state store. |
PROM_URL | — | Base URL of the Prometheus HTTP API. |
rollback_cooldown_s | 3600 | Minimum seconds before re-canarying a rolled-back fingerprint. |
Tune the step schedule and observation window together, not independently: a longer schedule with shorter windows and a shorter schedule with longer windows carry very different exposure, and mixing changes in one deploy makes the exposure impossible to reason about against replayed traffic. When a policy first WARNs and only later escalates to a hard block, hand that state machine to the escalation runbook rather than encoding it in the canary loop — see Implementing Warn-Then-Block Gate Escalation Policies.
Related
- Implementing Warn-Then-Block Gate Escalation Policies — the staged policy that annotates and admits a first breach and only blocks on repeated or worsening ones.
- Enforcing Plan Baselines with Argo CD — the GitOps deploy controller this gate emits its promote/rollback decisions into.
- Wiring Plan Regression Gates into GitHub Actions — the pull-request-time counterpart that blocks before traffic is ever routed.
- Tuning Thresholds for False Positive Reduction — the upstream evaluator whose scored verdict is the precondition for entering a canary.
- Regression Detection & Rule Engines — the offline scoring subsystem that classifies a plan before any live traffic sees it.