Enforcing Query Plan Baselines with Argo CD and Kubernetes

This guide is the GitOps enforcement stage that stops a Kubernetes manifest from shipping a regressed query plan: it consumes an already-scored plan verdict and turns it into an Argo CD sync-gate decision, so a rollout that would deploy a degraded plan is blocked before any Pod reaches the Kubernetes cluster. It covers the verdict contract the gate reads, a runnable PreSync hook Job, an Argo Rollouts AnalysisTemplate that gates a canary on a Prometheus metric, and the exact latency and success-condition numerics the gate is held to.

The stage owns exactly one decision boundary: given a verdict keyed by the plan fingerprint of the release under sync, does Argo CD proceed to apply the desired state or does it fail the sync operation. It never re-scores plans, never captures EXPLAIN output, and never queries the production optimizer. It reads a verdict that the regression detection rule engines already produced and translates it into a hook exit code and an Application sync status. Getting this boundary right is what separates a gate that safely fails closed from one that lets a regression self-heal past it.

Architectural Boundaries #

Upstream, the gate consumes a scored verdict from the threshold-tuning stage, delivered as a row in a verdict store (PostgreSQL) keyed by the deterministic plan fingerprint. The verdict is one of ALLOW, WARN, or BLOCK. The gate treats that store as read-only during the sync path — it never mutates a verdict to force a passing sync, because a gate that can rewrite its own input is not a gate.

Downstream, the gate emits its result through one of two Argo surfaces. In a plain Argo CD flow it emits through a PreSync resource-hook Job: a zero exit code lets the sync wave advance, a non-zero exit code fails the hook and therefore the whole sync operation, leaving the Application in OperationState Failed. In a progressive-delivery flow it emits through an Argo Rollouts AnalysisTemplate that queries a Prometheus counter and marks the analysis Successful or Failed, which the Rollout controller reads to promote or abort the canary. Either way the release manifest is blocked before it becomes live desired state.

The gate cross-references the same structural signals the scoring engines use — a BLOCK that coincides with an index-usage regression is escalated to a hard abort rather than a soft warning — but it does not recompute them. Its contract is deliberately narrow so it stays deterministic under retries: the same verdict row and the same manifest revision must always yield the same sync decision.

Any shortcut that breaks this boundary — a hook Job whose hook-failed delete policy removes it before Argo reads the exit status, or an auto-sync loop that re-applies desired state while the gate is still deciding — reintroduces exactly the silent rollout of a regressed plan the gate exists to prevent.

Deterministic Routing and Schema Enforcement #

The gate is a pure function of two inputs: the plan fingerprint carried on the release (an annotation on the Application or an image label) and the verdict row that fingerprint resolves to. It rejects a missing or malformed verdict rather than defaulting to ALLOW, because a silent default is indistinguishable from an approval and is the most common way a regression slips through a GitOps gate.

The verdict the hook reads is validated against this JSON Schema before it is allowed to influence the exit code:

{
  "$id": "https://queryplan.org/schemas/plan-sync-verdict.json",
  "type": "object",
  "additionalProperties": false,
  "required": ["plan_fingerprint", "verdict", "score", "evaluated_at"],
  "properties": {
    "plan_fingerprint": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
    "verdict":          { "type": "string", "enum": ["ALLOW", "WARN", "BLOCK"] },
    "score":            { "type": "number", "minimum": 0.0, "maximum": 1.0 },
    "evaluated_at":     { "type": "string", "format": "date-time" },
    "signals":          { "type": "array", "items": { "type": "string" } }
  }
}

The mapping from verdict to hook exit code is fixed and total — every verdict, and every failure to obtain one, resolves to a defined exit code:

Because the fingerprint is a 64-character SHA-256 string, the gate shards verdict lookups across read replicas with a stable key derived directly from it, so retries of the same sync always hit the same replica and read the same verdict:

replica_index = int(plan_fingerprint[:8], 16) % VERDICT_REPLICA_COUNT

A verdict is considered stale — and therefore fails closed — when evaluated_at is older than VERDICT_MAX_AGE_SECONDS (default 900). This prevents an old ALLOW from a superseded baseline from authorizing a sync of a manifest whose plan was never re-scored.

Production-Ready Implementation #

The hook below is the container entrypoint for an Argo CD PreSync Job. It resolves the plan fingerprint from the environment (injected by Argo from the Application annotation), reads the verdict through an asyncpg pool, and exits with the mapped code. Every path is instrumented with OpenTelemetry spans and metrics and logged through structlog. The critical property is that every error path — timeout, missing row, schema failure — exits non-zero, so the gate can only fail closed.

from __future__ import annotations

import asyncio
import os
import sys
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Literal

import asyncpg
import structlog
from opentelemetry import metrics, trace

log = structlog.get_logger("argocd.plan_gate")
tracer = trace.get_tracer("argocd.plan_gate")
meter = metrics.get_meter("argocd.plan_gate")

GATE_DURATION = meter.create_histogram("plan_sync_gate_duration_ms", unit="ms")
GATE_DECISION = meter.create_counter("plan_sync_gate_decision_total", unit="1")

VERDICT_MAX_AGE_SECONDS = int(os.environ.get("VERDICT_MAX_AGE_SECONDS", "900"))
LOOKUP_TIMEOUT_S = float(os.environ.get("VERDICT_LOOKUP_TIMEOUT_S", "3.0"))
EXIT_ALLOW = 0
EXIT_BLOCK = 1  # any non-zero fails the PreSync hook and aborts the Argo sync


@dataclass(frozen=True)
class SyncVerdict:
    plan_fingerprint: str
    verdict: Literal["ALLOW", "WARN", "BLOCK"]
    score: float
    evaluated_at: datetime


async def _fetch_verdict(pool: asyncpg.Pool, fingerprint: str) -> SyncVerdict | None:
    row = await pool.fetchrow(
        """
        SELECT plan_fingerprint, verdict, score, evaluated_at
        FROM plan_sync_verdict
        WHERE plan_fingerprint = $1
        ORDER BY evaluated_at DESC
        LIMIT 1
        """,
        fingerprint,
    )
    if row is None:
        return None
    return SyncVerdict(
        plan_fingerprint=row["plan_fingerprint"],
        verdict=row["verdict"],
        score=float(row["score"]),
        evaluated_at=row["evaluated_at"],
    )


def _decide(verdict: SyncVerdict) -> int:
    age = (datetime.now(timezone.utc) - verdict.evaluated_at).total_seconds()
    if age > VERDICT_MAX_AGE_SECONDS:
        log.error("verdict_stale", fingerprint=verdict.plan_fingerprint, age_s=age)
        return EXIT_BLOCK
    if verdict.verdict == "BLOCK":
        return EXIT_BLOCK
    return EXIT_ALLOW  # ALLOW and WARN both proceed; WARN is annotated upstream


async def run_gate() -> int:
    fingerprint = os.environ.get("PLAN_FINGERPRINT", "").strip()
    if len(fingerprint) != 64:
        log.error("missing_fingerprint", value=fingerprint[:12])
        return EXIT_BLOCK  # fail closed on a malformed release annotation

    started = asyncio.get_event_loop().time()
    with tracer.start_as_current_span("plan_sync_gate") as span:
        span.set_attribute("plan.fingerprint", fingerprint)
        pool: asyncpg.Pool | None = None
        try:
            pool = await asyncpg.create_pool(
                dsn=os.environ["VERDICT_DSN"],
                min_size=1,
                max_size=2,
                command_timeout=LOOKUP_TIMEOUT_S,
            )
            verdict = await asyncio.wait_for(
                _fetch_verdict(pool, fingerprint), timeout=LOOKUP_TIMEOUT_S
            )
            if verdict is None:
                log.error("verdict_absent", fingerprint=fingerprint)
                GATE_DECISION.add(1, {"decision": "BLOCK", "reason": "absent"})
                return EXIT_BLOCK

            code = _decide(verdict)
            decision = "ALLOW" if code == EXIT_ALLOW else "BLOCK"
            span.set_attribute("gate.decision", decision)
            span.set_attribute("gate.verdict", verdict.verdict)
            GATE_DECISION.add(1, {"decision": decision, "verdict": verdict.verdict})
            log.info(
                "plan_gate_decision",
                fingerprint=fingerprint,
                verdict=verdict.verdict,
                score=verdict.score,
                exit_code=code,
            )
            return code
        except (asyncio.TimeoutError, asyncpg.PostgresError, OSError) as exc:
            span.record_exception(exc)
            GATE_DECISION.add(1, {"decision": "BLOCK", "reason": "store_error"})
            log.error("verdict_store_unreachable", fingerprint=fingerprint, error=str(exc))
            return EXIT_BLOCK  # fail closed: never let a store outage open the gate
        finally:
            GATE_DURATION.record((asyncio.get_event_loop().time() - started) * 1000.0)
            if pool is not None:
                await pool.close()


if __name__ == "__main__":
    sys.exit(asyncio.run(run_gate()))

The Kubernetes manifest that runs this entrypoint as an Argo CD PreSync hook must set the deletion policy so the Job survives long enough for Argo to read its exit status. The hook-succeeded policy — not hook-failed — is what keeps a failed Job around for diagnosis:

apiVersion: batch/v1
kind: Job
metadata:
  name: plan-verdict-gate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 0                 # a retried Job can mask a BLOCK; never retry the gate
  activeDeadlineSeconds: 30
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: plan-gate-reader
      containers:
        - name: gate
          image: registry.internal/plan-verdict-gate:1.4.2
          env:
            - name: PLAN_FINGERPRINT
              value: "$ARGOCD_APP_REVISION_FINGERPRINT"
            - name: VERDICT_DSN
              valueFrom:
                secretKeyRef:
                  name: verdict-store-dsn
                  key: dsn
            - name: VERDICT_MAX_AGE_SECONDS
              value: "900"

For a progressive-delivery flow, the equivalent gate is an Argo Rollouts AnalysisTemplate that queries a Prometheus counter recording BLOCK decisions for the canary’s fingerprint. The analysis fails the moment a single BLOCK is observed, which aborts the canary:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: plan-verdict-analysis
spec:
  args:
    - name: fingerprint
  metrics:
    - name: plan-block-decisions
      interval: 30s
      count: 6
      failureLimit: 0            # one BLOCK aborts; do not tolerate any
      successCondition: result == 0
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: >
            sum(plan_sync_gate_decision_total{
              decision="BLOCK", fingerprint="{{args.fingerprint}}"})

Threshold Reference #

The gate carries its own operational SLOs, distinct from the verdicts it enforces. These numbers define when the gate itself is the problem — a slow gate that times out and fails closed will block healthy releases just as surely as a real regression.

A sustained fail-closed rate above 1 % almost always means the verdict store or its network path is degraded, not that the fleet regressed — the gate is correctly refusing to open, but the root cause is infrastructure, not plans. Wire that as a Prometheus alert so the on-call gets the gate itself, not a wall of blocked syncs:

groups:
  - name: plan-sync-gate
    rules:
      - alert: PlanGateFailingClosed
        expr: |
          (
            sum(rate(plan_sync_gate_decision_total{reason="store_error"}[10m]))
            /
            sum(rate(plan_sync_gate_decision_total[10m]))
          ) > 0.01
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Plan sync gate failing closed >1% — verdict store or network degraded"
          runbook: "Check verdict-store DSN health and asyncpg pool before disabling the gate."

      - alert: PlanGateLatencyHigh
        expr: histogram_quantile(0.95, sum(rate(plan_sync_gate_duration_ms_bucket[5m])) by (le)) > 1200
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "Gate p95 >1200ms — approaching activeDeadlineSeconds, syncs may abort"

Failure Scenarios and Root Cause Analysis #

1. PreSync hook not blocking because of hook-failed deletion policy. Symptom: a BLOCK verdict is logged by the Job, yet the Application still syncs Healthy. Root cause: the hook Job carries argocd.argoproj.io/hook-delete-policy: HookFailed, so Argo deletes the Job the instant it fails and loses the non-zero exit status, treating the hook as absent. Diagnose by inspecting the sync operation and confirming the Job vanished:

argocd app get plan-service --show-operation
kubectl get jobs -l argocd.argoproj.io/instance=plan-service --show-labels

Mitigation: set the delete policy to HookSucceeded (or BeforeHookCreation) so a failed Job is retained and its exit code is read; never use HookFailed on a gate.

2. AnalysisTemplate failureLimit set too high. Symptom: a canary carrying a BLOCK fingerprint promotes to stable anyway. Root cause: failureLimit is greater than 0, so the analysis tolerates one or more BLOCK measurements before failing, and the canary window closes first. Diagnose the analysis run:

kubectl argo rollouts get rollout plan-service --watch
kubectl get analysisrun -o jsonpath='{.items[*].status.metricResults[*].failed}'

Mitigation: set failureLimit: 0 and successCondition: result == 0, so a single BLOCK observation aborts the rollout immediately.

3. RBAC denying the hook Job database access. Symptom: every gate run fails closed with verdict_store_unreachable, blocking all syncs. Root cause: the plan-gate-reader ServiceAccount lacks a binding to the secret holding VERDICT_DSN, or a NetworkPolicy blocks egress to the verdict store. Diagnose the ServiceAccount’s access and the Pod logs:

kubectl auth can-i get secret/verdict-store-dsn --as=system:serviceaccount:apps:plan-gate-reader
kubectl logs job/plan-verdict-gate | grep verdict_store_unreachable

Mitigation: bind a least-privilege Role granting get on the DSN secret only, and add a NetworkPolicy egress rule to the verdict store’s port; keep the DB role read-only on plan_sync_verdict.

4. Verdict-store timeout resolving to fail-open by accident. Symptom: during a store brownout, syncs proceed instead of aborting. Root cause: a custom hook wrapper caught the timeout and exited 0, or command_timeout was unset so the Job hit activeDeadlineSeconds and Argo recorded the Job as deleted rather than failed. Diagnose:

kubectl describe job plan-verdict-gate | grep -E "DeadlineExceeded|Failed"
argocd app get plan-service -o json | jq '.status.operationState.phase'

Mitigation: ensure every exception path returns EXIT_BLOCK as in the reference hook, keep command_timeout below activeDeadlineSeconds, and set backoffLimit: 0 so a retry cannot convert a timeout into an eventual success.

5. Auto-sync racing the gate. Symptom: a manifest that the gate blocked reappears in the live cluster minutes later. Root cause: the Application has automated.selfHeal: true, so Argo re-applies desired state on the next reconcile, and if the PreSync Job is skipped on a self-heal path the gate never runs. Diagnose the sync policy and reconcile history:

argocd app get plan-service -o json | jq '.spec.syncPolicy'
argocd app history plan-service

Mitigation: keep the gate on a PreSync wave with sync-wave: "-1" so it precedes every apply including self-heal, and consider disabling selfHeal for gated Applications so a blocked revision cannot be silently reasserted. This interaction is covered in depth in the runbook for gating syncs on verdicts.

Configuration Reference #

Tighten VERDICT_MAX_AGE_SECONDS and VERDICT_LOOKUP_TIMEOUT_S together only when the verdict store is provably fast; loosening the timeout to survive a slow store trades sync latency for a wider window in which a stale verdict can authorize a release. Never raise failureLimit or backoffLimit to make a flaky gate “pass” — that converts a safety control into decoration.

Related #

• Tuning Thresholds for False Positive Reduction — the upstream scorer that produces the ALLOW/WARN/BLOCK verdict this gate reads.
• Gating Argo CD Syncs on Query Plan Verdicts — the step-by-step runbook for when a sync goes Healthy despite a BLOCK.
• Wiring Plan Regression Gates into GitLab CI — the pipeline-native counterpart for teams gating before the GitOps handoff.
• Designing Canary and Progressive Delivery Gates — how the AnalysisTemplate metric fits a broader canary strategy.
• Monitoring Index Usage Changes for Regression Signals — the structural signal that escalates a WARN to a hard abort.

← Back to CI/CD Gate Integration for Query Plan Regression