Gating Argo CD Syncs on Query Plan Verdicts

This runbook halts an Argo CD sync when the plan verdict is BLOCK and diagnoses the three ways a sync slips past that gate: an Application that reports Healthy despite a BLOCK, a PreSync hook that returns exit 0 when it should return non-zero, and an auto-sync loop that self-heals a blocked revision back into the live cluster. It gives the exact symptoms, the resource-deletion and sync-policy root causes, and the manifests, hook code, and argocd CLI checks to close each gap.

It is the operational companion to Enforcing Query Plan Baselines with Argo CD and Kubernetes, which defines the verdict contract and the hook implementation; this page assumes that contract and focuses on why a correctly-scored BLOCK still fails to stop a rollout.

Symptom Identification and Production Thresholds #

Each of the following is a hard breach condition, not a warning sign. When any one fires, the gate is not enforcing and a regressed plan can reach production.

1. Application syncs Healthy despite a BLOCK verdict. The verdict store holds a fresh BLOCK row (evaluated_at within 900 seconds) for the synced revision’s fingerprint, yet argocd app get reports Sync Status: Synced and Health Status: Healthy. Expected behavior is OperationState: Failed. Any occurrence is a Sev-2.
2. PreSync hook returns exit 0 on a BLOCK. The gate Job’s logs show plan_gate_decision ... verdict=BLOCK, but the Pod’s terminated state reports exitCode: 0. The exit code and the logged verdict disagree — the mapping is broken.
3. Auto-sync self-heals past the gate. A revision that was aborted at T0 reappears as live desired state by T0 + reconcile_interval (default 180 seconds) with no PreSync Job in its history, because a self-heal reconcile skipped the hook.
4. Gate fail-open on store error. More than 0 syncs proceed while the gate Job logs verdict_store_unreachable. A store outage must fail closed; a single fail-open sync is a breach.
5. Stale ALLOW authorizing a re-scored plan. A sync proceeds on an ALLOW whose evaluated_at is older than 900 seconds while a newer BLOCK exists for the same fingerprint — the gate read the wrong row.

Root Cause Analysis #

Four failure domains produce those symptoms. Isolate which one you are in before touching any manifest.

Hook resource deletion policy. Argo CD reads a hook’s outcome from the hook resource’s final status. If the Job carries argocd.argoproj.io/hook-delete-policy: HookFailed, Argo deletes the Job the moment it fails and can no longer observe the non-zero exit, so the hook is treated as if it never ran and the sync proceeds. Confirm the annotation on the live resource:

kubectl get job plan-verdict-gate -o jsonpath='{.metadata.annotations}' | tr ',' '\n'

If the output contains "hook-delete-policy":"HookFailed", that is your bug — it maps directly to symptom 1.

sync-options suppressing hook failure. The Application may set sync options that change how a failed hook is treated. SkipHooks=true disables hooks entirely; a PrunePropagationPolicy or a mis-scoped syncOptions list can also drop the gate wave. Inspect the effective options:

argocd app get plan-service -o json | jq '.spec.syncPolicy.syncOptions'

Any entry that disables or skips hooks means the gate never executes, producing symptoms 1 and 2.

Auto-sync with selfHeal. When syncPolicy.automated.selfHeal is true, Argo continuously reconciles live state toward Git. A revision you aborted is still the Git HEAD, so the next reconcile re-applies it; if that reconcile path does not re-run the PreSync wave, the block is bypassed. This is symptom 3. Check the policy:

argocd app get plan-service -o json | jq '.spec.syncPolicy.automated'

A result of {"prune": true, "selfHeal": true} on a gated Application is the root cause.

RBAC / verdict resolution. If the gate Job cannot read the verdict store, a buggy wrapper that swallows the error and exits 0 yields fail-open (symptom 4); if the query orders rows wrong it can read a stale ALLOW (symptom 5). Verify the ServiceAccount and the row the gate would read:

kubectl auth can-i get secret/verdict-store-dsn \
  --as=system:serviceaccount:apps:plan-gate-reader

Step-by-Step Remediation #

Apply the fixes for the domain you isolated. Each step includes the manifest or command and the output that confirms it worked.

1. Correct the hook deletion policy. Change the Job so a failed gate is retained and its exit code is read:

apiVersion: batch/v1
kind: Job
metadata:
  name: plan-verdict-gate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded   # not HookFailed
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 30
  template:
    spec:
      restartPolicy: Never
      serviceAccountName: plan-gate-reader
      containers:
        - name: gate
          image: registry.internal/plan-verdict-gate:1.4.2

2. Make the hook exit code match the verdict. The mapping must be total and every error path must fail closed:

EXIT_ALLOW, EXIT_BLOCK = 0, 1

def gate_exit_code(verdict: str | None, age_s: float, max_age_s: int = 900) -> int:
    if verdict is None or age_s > max_age_s:
        return EXIT_BLOCK          # absent or stale → fail closed
    return EXIT_BLOCK if verdict == "BLOCK" else EXIT_ALLOW

3. Remove hook-skipping sync options and pin auto-sync. Strip SkipHooks and disable selfHeal for the gated Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: plan-service
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false            # a blocked revision must not be reasserted
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
      # SkipHooks removed — the gate must run on every sync

4. Force a re-sync and read the operation state. Trigger the sync and confirm the gate now blocks:

argocd app sync plan-service --prune
argocd app get plan-service -o json | jq '.status.operationState.phase'

Expected output when the gate correctly blocks a BLOCK fingerprint:

"Failed"

And the gate Job’s terminated state should agree with the logged verdict:

kubectl get pod -l job-name=plan-verdict-gate \
  -o jsonpath='{.items[0].status.containerStatuses[0].state.terminated.exitCode}'

1

Verification Checklist #

• [ ] The gate Job annotation is hook-delete-policy: HookSucceeded, never HookFailed.
• [ ] A synthetic BLOCK fingerprint drives argocd app get to OperationState: Failed.
• [ ] The gate Pod’s terminated.exitCode is 1 whenever its logs show verdict=BLOCK.
• [ ] spec.syncPolicy.syncOptions contains no SkipHooks=true entry.
• [ ] spec.syncPolicy.automated.selfHeal is false on every gated Application.
• [ ] A verdict-store outage produces Failed syncs (fail closed), with 0 proceeding.
• [ ] A stale ALLOW older than 900 seconds no longer authorizes a sync when a newer BLOCK exists.
• [ ] The gate runs on sync-wave: "-1" so it precedes every workload apply, including self-heal.

Compatibility and Engine-Specific Notes #

The gate mechanics differ across the delivery controllers you might run. Map the concept before wiring it.

For an app-of-apps layout, the gate belongs on each child Application, not the root — a root-level PreSync runs once and cannot see per-service fingerprints. When ordering matters across services, use ascending sync-wave values so the gate wave (-1) always precedes the workload wave (0) within each Application, and keep every gated child on selfHeal: false. Argo Rollouts users should prefer the AnalysisTemplate path with autoPromotionEnabled: false so a canary cannot promote before the analysis observes a BLOCK; this pairs with the broader strategy in Designing Canary and Progressive Delivery Gates.

Related #

• Enforcing Query Plan Baselines with Argo CD and Kubernetes — the parent guide defining the verdict contract, hook implementation, and SLOs.
• Wiring Plan Regression Gates into GitLab CI — the pipeline-side gate that scores a plan before the GitOps handoff.
• Designing Canary and Progressive Delivery Gates — the Argo Rollouts AnalysisTemplate approach in a full canary flow.
• Monitoring Index Usage Changes for Regression Signals — the structural signal that upgrades a WARN to a BLOCK.

← Back to Enforcing Query Plan Baselines with Argo CD and Kubernetes