Implementing Warn-Then-Block Gate Escalation Policies

A warn-then-block policy admits a first regression with an annotation and an acknowledgement path, then escalates the same fingerprint to a hard block only after repeated or worsening breaches. This runbook shows why a stateless gate can never make that escalation fire, how to give it a durable escalation ledger with time decay, and the exact counting logic, configuration, and verification steps that turn a stream of ignorable WARNs into one decisive BLOCK.

The failure this fixes is specific: teams soften a hard gate to a warning to stop blocking unrelated pull requests, and then every regression WARNs forever and nothing ever blocks. The policy below keeps the soft first touch but restores teeth — a fingerprint that breaches three times inside the decay window, or breaches worse than it did before, is blocked automatically. It is the escalation half of the canary and progressive-delivery gate, and it consumes the same scored verdict produced by Tuning Thresholds for False Positive Reduction.

Symptom Identification and Production Thresholds #

Treat each of the following as a hard trigger that the escalation policy is broken, not a matter of taste:

1. Same fingerprint WARNs without escalating. One query_hash produces a WARN verdict 8 or more times across a rolling 14-day window and never once emits BLOCK. A recurring breach that never escalates means the gate has no memory of the prior breaches.
2. Breach counter stuck at 1. The ledger row for a repeatedly-breaching fingerprint shows breach_count = 1 and a first_seen timestamp that keeps advancing to match last_seen. The count is being overwritten, not incremented.
3. WARN fatigue. The acknowledgement rate on WARN annotations exceeds 75% and the median time-to-ack climbs past 6 hours. When almost every WARN is rubber-stamped, the annotation has stopped carrying signal and the policy must escalate on repetition instead of relying on human attention.
4. Escalation never fires above the score threshold. The computed weighted_score for a fingerprint reaches or exceeds the escalation threshold of 3.0, yet the emitted verdict is still WARN. The score is correct but the decision branch that reads it is missing or unreachable.
5. Ack resets the counter. A verdict flips from an escalating state back to WARN immediately after an operator acknowledges the alert, and breach_count drops to 0 in the same transaction. Acknowledgement is silencing the counter instead of only silencing the notification.

Root Cause Analysis #

Three failure domains produce every symptom above. Each has a direct diagnostic.

The gate is stateless and cannot count occurrences. A CI job evaluates one pull request in isolation and exits; nothing persists between runs, so the third breach looks identical to the first. Confirm there is no durable memory by checking whether any ledger table exists at all:

SELECT to_regclass('public.gate_escalation_ledger') AS ledger_table;
-- NULL means the gate has nowhere to remember prior breaches.

If the result is NULL, escalation is structurally impossible — the gate is re-deciding from zero every run. This is the same statelessness that makes an offline verdict replayable, but here it is the bug: escalation is inherently stateful and must live outside the per-run process.

The escalation ledger is missing or unkeyed. Even with a table present, escalation fails if rows are keyed by CI run ID or commit SHA rather than by query_hash, because every commit then starts a fresh count. Inspect the effective key and whether counts ever exceed one:

SELECT query_hash, breach_count, first_seen, last_seen
FROM gate_escalation_ledger
ORDER BY last_seen DESC
LIMIT 20;

A table where breach_count never exceeds 1 while last_seen keeps moving is the signature of upsert logic that overwrites instead of incrementing — usually an INSERT ... ON CONFLICT DO UPDATE SET breach_count = 1 where the intended clause was breach_count = ledger.breach_count + 1.

Acknowledgement resets the counter. The ack handler is meant to quiet the notification, but a well-intentioned UPDATE ... SET breach_count = 0 on ack erases the history the policy depends on. Find ack events that zeroed a live count:

SELECT query_hash, breach_count, acked_until, weighted_score
FROM gate_escalation_ledger
WHERE acked_until > now() AND breach_count = 0 AND weighted_score = 0;

Any row here was silenced and forgotten in the same step. Acknowledgement must set acked_until only; the count and the decayed score must survive it untouched.

Step-by-Step Remediation #

1. Create the escalation ledger #

The ledger is one durable row per fingerprint. It stores the running count, a time-decayed weighted score, the last severity seen, and an acknowledgement window that suppresses notifications without touching the count.

CREATE TABLE gate_escalation_ledger (
    query_hash      char(64) PRIMARY KEY,
    breach_count    integer      NOT NULL DEFAULT 0,
    weighted_score  double precision NOT NULL DEFAULT 0.0,
    last_severity   double precision NOT NULL DEFAULT 0.0,
    first_seen      timestamptz  NOT NULL DEFAULT now(),
    last_seen       timestamptz  NOT NULL DEFAULT now(),
    acked_until     timestamptz,
    state           text NOT NULL DEFAULT 'WARN'
                    CHECK (state IN ('WARN', 'ESCALATING', 'BLOCK'))
);
CREATE INDEX ON gate_escalation_ledger (last_seen);

2. Implement the counting and decay logic #

The decay is what stops a breach from a month ago from blocking today’s deploy. Each new breach decays the prior score by a 7-day half-life before adding the current severity, so only breaches that cluster in time accumulate. Escalation fires when the decayed score reaches 3.0 or when the current breach is strictly worse than the last one seen.

from __future__ import annotations

import math
from dataclasses import dataclass
from datetime import datetime, timezone

import asyncpg
import structlog
from opentelemetry import trace

log = structlog.get_logger(__name__)
tracer = trace.get_tracer(__name__)

HALF_LIFE_S = 7 * 24 * 3600          # 7-day decay half-life
ESCALATE_SCORE = 3.0                 # weighted-score block threshold
ESCALATE_COUNT = 3                   # raw-count block threshold
ACK_SUPPRESS_S = 24 * 3600           # notification silence, never a reset


@dataclass(frozen=True)
class Breach:
    query_hash: str
    severity: float                  # normalized cost/latency overshoot, e.g. 1.4 == 40% over


def _decay(prior_score: float, elapsed_s: float) -> float:
    return prior_score * math.pow(0.5, elapsed_s / HALF_LIFE_S)


async def record_and_decide(pool: asyncpg.Pool, breach: Breach) -> str:
    with tracer.start_as_current_span("escalation.record") as span:
        span.set_attribute("query_hash", breach.query_hash)
        async with pool.acquire() as conn:
            async with conn.transaction():
                row = await conn.fetchrow(
                    "SELECT * FROM gate_escalation_ledger WHERE query_hash = $1 FOR UPDATE",
                    breach.query_hash,
                )
                now = datetime.now(timezone.utc)
                if row is None:
                    decayed = 0.0
                    count = 1
                    worsened = False
                else:
                    elapsed = (now - row["last_seen"]).total_seconds()
                    decayed = _decay(row["weighted_score"], elapsed)
                    count = row["breach_count"] + 1
                    worsened = breach.severity > row["last_severity"]

                score = decayed + breach.severity
                escalate = score >= ESCALATE_SCORE or count >= ESCALATE_COUNT or worsened
                state = "BLOCK" if escalate else "WARN"

                await conn.execute(
                    """
                    INSERT INTO gate_escalation_ledger
                        (query_hash, breach_count, weighted_score, last_severity, last_seen, state)
                    VALUES ($1, $2, $3, $4, now(), $5)
                    ON CONFLICT (query_hash) DO UPDATE SET
                        breach_count   = $2,
                        weighted_score = $3,
                        last_severity  = $4,
                        last_seen      = now(),
                        state          = $5
                    """,
                    breach.query_hash, count, round(score, 4), breach.severity, state,
                )
                span.set_attribute("weighted_score", round(score, 4))
                span.set_attribute("decision", state)
                log.info(
                    "escalation_decision",
                    query_hash=breach.query_hash,
                    breach_count=count,
                    weighted_score=round(score, 4),
                    severity=breach.severity,
                    worsened=worsened,
                    decision=state,
                )
                return state

Acknowledgement is a separate call that sets acked_until only — it never touches breach_count or weighted_score, which is exactly the bug from the third failure domain:

async def acknowledge(pool: asyncpg.Pool, query_hash: str) -> None:
    await pool.execute(
        """
        UPDATE gate_escalation_ledger
        SET acked_until = now() + make_interval(secs => $2)
        WHERE query_hash = $1
        """,
        query_hash, ACK_SUPPRESS_S,
    )

3. Bind the thresholds in config #

Keep the policy numbers in one file so CI, the ledger service, and the deploy gate read identical values:

escalation:
  half_life_hours: 168        # 7-day decay
  escalate_score: 3.0         # weighted-score block threshold
  escalate_count: 3           # raw repeat-count block threshold
  ack_suppress_hours: 24      # notification silence only, never a counter reset
  worsen_blocks: true         # a strictly-worse breach blocks immediately
  warn_annotation: "Plan regression WARN — acknowledge to admit; repeats will block."

4. Confirm the expected output #

A third clustered breach on the same fingerprint crosses the score threshold and emits a single structured line, and the deploy job reads decision=BLOCK from it:

2026-07-18T09:14:52Z [info] escalation_decision query_hash=7c2b…e1 breach_count=3 weighted_score=3.42 severity=1.35 worsened=false decision=BLOCK

Verification Checklist #

• [ ] gate_escalation_ledger exists and is keyed on query_hash, not commit SHA or run ID.
• [ ] A fingerprint that breaches three times inside 7 days emits decision=BLOCK on the third breach.
• [ ] breach_count strictly increments across runs and is never overwritten to 1.
• [ ] Acknowledging a WARN sets acked_until and leaves breach_count and weighted_score unchanged.
• [ ] A breach older than two half-lives (14 days) decays below 1.0 and no longer contributes to a block on its own.
• [ ] A single breach whose severity exceeds the previous last_severity escalates immediately when worsen_blocks is true.
• [ ] The escalate_score and half_life_hours values in the running service match the YAML config exactly.
• [ ] WARN annotations carry the acknowledgement instructions and a link back to the fingerprint’s ledger row.

Compatibility and Engine-Specific Notes #

The ledger and decay logic are CI-agnostic; only the mechanism that turns a BLOCK decision into a red pipeline differs per system.

On all three, the ledger must live outside the runner or controller — an ephemeral runner has no memory, so a ledger stored in job state resets the count every run and reproduces the original stateless bug. Wire the escalation read as a required check on the protected branch so a BLOCK cannot be merged past, and route the same decision into the canary and progressive-delivery gate so a fingerprint that has escalated to BLOCK never enters a live rollout. For the GitOps and pull-request wiring specifics, pair this with Enforcing Plan Baselines with Argo CD and Wiring Plan Regression Gates into GitHub Actions.

Related #

• Tuning Thresholds for False Positive Reduction — the evaluator whose WARN verdict is the first breach this policy records.
• Enforcing Plan Baselines with Argo CD — how a BLOCK decision halts a GitOps rollout.
• Wiring Plan Regression Gates into GitHub Actions — turning the escalation read into a required status check.
• Regression Detection & Rule Engines — the scoring subsystem that produces the severities this ledger accumulates.

← Back to Designing Canary and Progressive-Delivery Gates for Plan Rollouts