Wiring Query Plan Regression Gates into GitLab CI

This guide wires the query-plan regression gate into a GitLab CI/CD pipeline as a dedicated plan-gate stage that turns an already-scored verdict into a hard pass/fail on the merge request. It covers the verdict contract the job consumes, the deterministic mapping from verdict band to job exit code, a full runnable asyncio gate job that posts a Merge Request note, and the exact latency SLOs and allow_failure policy that keep the gate both fast and safe.

The gate is deliberately dumb about why a plan regressed. All statistical judgement happens upstream in the regression detection rule engines, which emit a single classified verdict per query fingerprint. This job consumes that verdict, decides whether the pipeline job succeeds or fails, and writes a human-readable note back onto the merge request. It never recomputes cost deltas, never opens a write transaction against the baseline store, and never captures a plan itself — so a failure here is a wiring or availability failure, never a scoring dispute.

Architectural Boundaries #

Upstream, the job reads a scored verdict produced by the threshold-tuning stage, which classifies each fingerprint into a PASS / WARN / FAIL band with a confidence score. The verdict is materialised in a Postgres verdict store keyed by the commit SHA and query fingerprint, so the gate does a keyed lookup rather than a live evaluation — the CI runner touches the database read-only through a short-lived asyncpg connection scoped to the job.

Downstream, the job emits exactly three artifacts. First, a pipeline job status: the process exits 0 to pass or 1 to fail, and GitLab propagates that to the pipeline and, through merge-request pipelines, to the MR’s mergeability. Second, a Merge Request note posted through the GitLab REST API summarising the worst verdict band, the offending fingerprints, and the cost multiplier. Third, an optional allow_failure signal: for WARN-band verdicts the job can be marked non-blocking so the pipeline stays green while still surfacing the note, whereas FAIL-band verdicts must never be allowed to fail silently.

The routing that decides whether a regression actually blocks the MR is documented in the companion runbook, Blocking Merge Requests on Plan Cost Regression in GitLab CI. The verdict store itself is populated by tracking cost deltas across baseline versions, which anchors each candidate plan against the promoted baseline.

Any deviation from this boundary — recomputing deltas inside the runner, or letting the job swallow a database error into a green exit — reintroduces exactly the class of silent-pass bugs the gate exists to prevent.

Deterministic Routing and Schema Enforcement #

The gate is a pure function of the verdict it reads plus the GitLab predefined variables in the job environment. The verdict contract is strict: the job rejects a malformed or missing row rather than defaulting to green, because a silently coerced verdict is the single most dangerous failure a CI gate can have.

The verdict row the job fetches is validated against this JSON Schema before the exit code is computed:

{
  "$id": "https://queryplan.org/schemas/gitlab-plan-verdict.json",
  "type": "object",
  "additionalProperties": false,
  "required": ["commit_sha", "worst_band", "max_cost_multiplier", "offending_fingerprints", "evaluated_at"],
  "properties": {
    "commit_sha":            { "type": "string", "pattern": "^[a-f0-9]{40}$" },
    "worst_band":            { "type": "string", "enum": ["PASS", "WARN", "FAIL"] },
    "max_cost_multiplier":   { "type": "number", "minimum": 0 },
    "offending_fingerprints":{ "type": "array", "items": { "type": "string", "pattern": "^[a-f0-9]{64}$" }, "maxItems": 100 },
    "evaluated_at":          { "type": "string", "format": "date-time" }
  }
}

Routing from worst_band to a GitLab job exit code is a fixed, ordered map — there is no arithmetic and no tolerance at gate time, because the tolerance was already applied upstream:

1. PASS — exit 0. The job succeeds cleanly and posts a one-line confirmation note only if a prior WARN/FAIL note exists for the SHA (to avoid noise).
2. WARN — exit 0, but the job is marked allow_failure only when BLOCK_ALLOW_FAILURE is false and the band is WARN. The note carries the max_cost_multiplier so reviewers see the drift without the pipeline turning red.
3. FAIL — exit 1 unconditionally. allow_failure must be false for this band; a FAIL that is allowed to fail is the highest-severity misconfiguration this system tracks.

Because the verdict store is sharded, the gate reads the row for the exact commit under test using a stable partition key derived from the SHA, so a job on a retried pipeline always reads the same shard:

partition_key = int(commit_sha[:8], 16) % VERDICT_SHARD_COUNT

This guarantees a retried job and its original read identical verdict rows, eliminating cross-shard verdict drift without any coordination between runners.

Production-Ready Implementation #

The job below is a single asyncio entry point suitable for a GitLab runner. It resolves the verdict through an asyncpg connection bounded by GATE_TIMEOUT_MS, posts the Merge Request note through the GitLab REST API with httpx, instruments the whole run with OpenTelemetry spans and metrics, and logs structured events through structlog. Every exit path is explicit; a verdict-store timeout fails closed with exit code 1.

from __future__ import annotations

import asyncio
import os
import sys
from dataclasses import dataclass
from typing import Literal

import asyncpg
import httpx
import structlog
from opentelemetry import metrics, trace

log = structlog.get_logger("plan_gate.gitlab")
tracer = trace.get_tracer("plan_gate.gitlab")
meter = metrics.get_meter("plan_gate.gitlab")

GATE_DURATION = meter.create_histogram("plan_gate_duration_ms", unit="ms")
GATE_OUTCOME = meter.create_counter("plan_gate_outcome_total", unit="1")

Band = Literal["PASS", "WARN", "FAIL"]
EXIT_PASS, EXIT_BLOCK = 0, 1


@dataclass(frozen=True)
class GateConfig:
    dsn: str
    gitlab_api: str
    gitlab_token: str
    project_id: str
    mr_iid: str
    commit_sha: str
    timeout_ms: int = 900
    block_allow_failure: bool = False


def load_config() -> GateConfig:
    return GateConfig(
        dsn=os.environ["PLAN_GATE_DSN"],
        gitlab_api=os.environ.get("CI_API_V4_URL", "https://gitlab.com/api/v4"),
        gitlab_token=os.environ["CI_GITLAB_TOKEN"],
        project_id=os.environ["CI_PROJECT_ID"],
        mr_iid=os.environ["CI_MERGE_REQUEST_IID"],
        commit_sha=os.environ["CI_COMMIT_SHA"],
        timeout_ms=int(os.environ.get("GATE_TIMEOUT_MS", "900")),
        block_allow_failure=os.environ.get("BLOCK_ALLOW_FAILURE", "false").lower() == "true",
    )


async def _fetch_verdict(cfg: GateConfig) -> asyncpg.Record | None:
    conn = await asyncpg.connect(dsn=cfg.dsn, command_timeout=cfg.timeout_ms / 1000.0)
    try:
        return await conn.fetchrow(
            """
            SELECT worst_band, max_cost_multiplier, offending_fingerprints
            FROM plan_verdicts
            WHERE commit_sha = $1
            ORDER BY evaluated_at DESC
            LIMIT 1
            """,
            cfg.commit_sha,
        )
    finally:
        await conn.close()


async def _post_mr_note(cfg: GateConfig, body: str) -> None:
    url = f"{cfg.gitlab_api}/projects/{cfg.project_id}/merge_requests/{cfg.mr_iid}/notes"
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.post(
            url,
            headers={"PRIVATE-TOKEN": cfg.gitlab_token},
            json={"body": body},
        )
        resp.raise_for_status()


def _render_note(band: Band, multiplier: float, fingerprints: list[str]) -> str:
    icon = {"PASS": "✅", "WARN": "⚠️", "FAIL": "⛔"}[band]
    head = f"{icon} **Plan gate: {band}** — worst cost multiplier `{multiplier:.2f}x`"
    if band == "PASS":
        return f"{head}\n\nNo blocking plan-cost regression detected."
    listed = "\n".join(f"- `{fp[:16]}`" for fp in fingerprints[:10])
    return f"{head}\n\nOffending query fingerprints:\n{listed}"


async def run_gate(cfg: GateConfig) -> int:
    with tracer.start_as_current_span("plan_gate.run") as span:
        span.set_attribute("commit_sha", cfg.commit_sha)
        loop = asyncio.get_running_loop()
        started = loop.time()
        try:
            row = await asyncio.wait_for(
                _fetch_verdict(cfg), timeout=cfg.timeout_ms / 1000.0
            )
        except (asyncio.TimeoutError, asyncpg.PostgresError, OSError) as exc:
            # Fail closed: an unreachable verdict store must block, never pass.
            span.record_exception(exc)
            GATE_OUTCOME.add(1, {"outcome": "fail_closed"})
            log.error("verdict_store_unavailable", commit_sha=cfg.commit_sha, error=str(exc))
            await _safe_note(cfg, "⛔ **Plan gate: fail-closed** — verdict store unreachable.")
            return EXIT_BLOCK
        finally:
            GATE_DURATION.record((loop.time() - started) * 1000.0)

        if row is None:
            # No verdict for this SHA — treat as a hard block, not a pass.
            span.set_attribute("verdict_missing", True)
            GATE_OUTCOME.add(1, {"outcome": "missing"})
            log.error("verdict_missing", commit_sha=cfg.commit_sha)
            await _safe_note(cfg, "⛔ **Plan gate: no verdict** — scoring did not run for this commit.")
            return EXIT_BLOCK

        band: Band = row["worst_band"]
        multiplier = float(row["max_cost_multiplier"])
        fingerprints = list(row["offending_fingerprints"])
        span.set_attribute("worst_band", band)
        span.set_attribute("max_cost_multiplier", multiplier)

        await _safe_note(cfg, _render_note(band, multiplier, fingerprints))
        GATE_OUTCOME.add(1, {"outcome": band})
        log.info("gate_decided", commit_sha=cfg.commit_sha, band=band, multiplier=multiplier)
        return EXIT_BLOCK if band == "FAIL" else EXIT_PASS


async def _safe_note(cfg: GateConfig, body: str) -> None:
    # A note failure must never change the gate outcome.
    try:
        await _post_mr_note(cfg, body)
    except httpx.HTTPError as exc:
        log.warning("mr_note_failed", error=str(exc))


def main() -> None:
    cfg = load_config()
    sys.exit(asyncio.run(run_gate(cfg)))


if __name__ == "__main__":
    main()

Two properties make this safe in production. First, every non-PASS/WARN path — timeout, missing row, corrupt band — returns EXIT_BLOCK, so the absence of a clean PASS is always a block, never a silent green. Second, the MR-note post is isolated behind _safe_note: a GitLab API hiccup degrades the reviewer experience but never flips the gate verdict.

The .gitlab-ci.yml that runs this job uses a dedicated stage and merge-request rules so the gate only executes on MR pipelines, and it wires allow_failure from a variable so policy lives in one place:

stages:
  - build
  - plan-gate

plan-gate:
  stage: plan-gate
  image: python:3.12-slim
  variables:
    GATE_TIMEOUT_MS: "900"
    BLOCK_ALLOW_FAILURE: "false"
  needs:
    - job: build
      artifacts: false
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: on_success
      allow_failure: false
  before_script:
    - pip install --no-cache-dir asyncpg httpx structlog opentelemetry-sdk
  script:
    - python -m plan_gate.gitlab

The rules: clause binds the gate to merge-request pipelines only, so it evaluates against the merge result rather than a stray branch commit, and needs: lets it start the moment build finishes rather than waiting on the whole prior stage.

Threshold Reference #

The gate carries operational SLOs independent of any verdict it emits. These are the numbers alerting is wired against — they define when the gate itself, not the code under review, is misbehaving.

The allow_failure policy is band-exact and non-negotiable: PASS runs with allow_failure: false (it exits 0 anyway); WARN may run with allow_failure: true only when BLOCK_ALLOW_FAILURE=false; FAIL must always run with allow_failure: false. A sustained fail-closed rate above 2 % means the verdict store or its network path is unhealthy, not that the fleet regressed — encode that as a Prometheus alert:

groups:
  - name: gitlab-plan-gate
    rules:
      - alert: PlanGateFailClosedStorm
        expr: |
          (
            sum(rate(plan_gate_outcome_total{outcome="fail_closed"}[10m]))
            /
            sum(rate(plan_gate_outcome_total[10m]))
          ) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Plan gate failing closed >2% for 10m — verdict store or network unhealthy"
          runbook: "Check plan_verdicts replica health and GATE_TIMEOUT_MS before re-running MR pipelines."

      - alert: PlanGateLatencyBudgetBurn
        expr: |
          histogram_quantile(0.95, sum(rate(plan_gate_duration_ms_bucket[5m])) by (le)) > 1500
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "Plan gate p95 above 1500 ms — over the 900 ms SLO budget"

Failure Scenarios and Root Cause Analysis #

1. allow_failure: true silently passing a BLOCK. Symptom: an MR merges with a FAIL verdict note attached and a green pipeline. Root cause: the plan-gate job (or its rules: entry) carries allow_failure: true, so GitLab records the exit-1 as a non-blocking failure. Diagnose by inspecting the resolved job config:

glab ci config compile | grep -A6 '^plan-gate:' | grep allow_failure

Mitigation: set allow_failure: false on the rules: entry for merge-request pipelines and gate the WARN-only allowance behind BLOCK_ALLOW_FAILURE=false in code, so a FAIL band can never inherit the non-blocking flag.

2. Detached merge-request pipeline not running the gate. Symptom: the MR shows a passing pipeline but no plan-gate job appears. Root cause: the pipeline was created as a plain branch pipeline ($CI_PIPELINE_SOURCE == "push"), and the gate’s rules: only match merge_request_event, so the job was never added. Diagnose:

glab api "projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/pipelines" \
  | jq -r '.[0] | "\(.source)\t\(.status)"'

Mitigation: require merge-request pipelines via workflow: rules so a branch push alone cannot satisfy the MR, and set “Pipelines must succeed” on the target branch.

3. Protected-variable unavailability. Symptom: the gate crashes with a KeyError on CI_GITLAB_TOKEN for MRs from forks or non-protected branches. Root cause: CI_GITLAB_TOKEN is a protected CI/CD variable and is not injected into pipelines running on unprotected refs. Diagnose by checking whether the ref is protected:

glab api "projects/$CI_PROJECT_ID/protected_branches" | jq -r '.[].name'

Mitigation: either restrict MR pipelines to protected branches, or supply the token through a group-level protected variable scoped to the environment, and fail closed with a clear log line rather than a raw KeyError.

4. Verdict-store timeout falling open. Symptom: during a database failover, MRs merge with no plan-gate note. Root cause: an earlier gate implementation caught the timeout and returned exit 0. Diagnose:

SELECT state, count(*) FROM pg_stat_activity
WHERE application_name = 'plan_gate_gitlab'
GROUP BY state;

Mitigation: confirm the timeout path returns EXIT_BLOCK (as in the reference above), route the read to a warm replica, and keep GATE_TIMEOUT_MS at 900 so a slow store trips the block within a bounded window instead of hanging the runner.

5. Pipelines-for-merged-results skipping the gate. Symptom: the gate runs on the source branch head but not against the merged result, so a regression introduced by the merge itself slips through. Root cause: the project uses merged-results pipelines but the gate’s rules: matched only the detached MR event. Diagnose by reading the pipeline’s merge_request ref:

glab api "projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID" | jq -r '.ref'

Mitigation: keep the merge_request_event rule (which covers both detached and merged-results pipelines) and enable merge trains so the final gate runs against the exact commit that will land on the target branch.

Configuration Reference #

Change GATE_TIMEOUT_MS and the replica routing when chasing latency, and change BLOCK_ALLOW_FAILURE only in a deliberate, reviewed step — never flip it to true to unstick a pipeline, because that is precisely how a FAIL verdict starts merging silently.

Related #

• Blocking Merge Requests on Plan Cost Regression in GitLab CI — the runbook for making an MR unmergeable when this gate reports a regression.
• Wiring Query Plan Regression Gates into GitHub Actions — the equivalent integration for GitHub’s check-run model.
• Enforcing Plan Baselines with Argo CD — extending the same verdict contract to GitOps deploy gates.
• Tracking Cost Deltas Across Baseline Versions — the upstream stage that populates the verdict store this gate reads.
• Tuning Thresholds for False Positive Reduction — how the PASS/WARN/FAIL band the gate consumes is decided.

← Back to CI/CD Gate Integration for Query Plan Regression