Blocking Merge Requests on Plan Cost Regression in GitLab CI

This runbook makes a GitLab merge request genuinely unmergeable when a query plan’s cost regresses past the gate’s threshold, closing the common gaps where a red-band verdict still lands on the target branch. It walks the exact symptoms of a leaky gate, the four configuration domains that let a regression through, and step-by-step remediation with .gitlab-ci.yml, Python, and glab commands you can paste directly. It assumes the plan-gate job from Wiring Query Plan Regression Gates into GitLab CI is already emitting verdicts and posting Merge Request notes.

The failure this runbook fixes is subtle: the gate runs, the note says FAIL, and the MR merges anyway. In GitLab that outcome is almost never a bug in the gate script — it is a mismatch between the job’s allow_failure flag, the branch’s merge settings, and which pipeline type actually gated the merge. Get those three aligned and a cost regression above the threshold becomes a hard stop.

Symptom Identification and Production Thresholds #

Treat each of the following as a breach condition with an exact trigger, not a guideline. Any one of them means the gate is reporting correctly but failing to block.

1. MR mergeable despite a cost delta over the threshold. The plan-gate note reads FAIL with max_cost_multiplier above 3.0x, yet the “Merge” button is enabled and the pipeline badge is green. The verdict fired but the pipeline status did not propagate to mergeability.
2. Job green because of allow_failure. The plan-gate job shows a yellow “allowed to fail” warning triangle rather than a red cross, and the overall pipeline status is passed. Exit code 1 was recorded as a non-blocking failure.
3. “Merge when pipeline succeeds” bypassing the gate. An MR set to auto-merge lands within seconds of the pipeline starting, before the plan-gate stage runs, because an earlier stage’s success was treated as sufficient.
4. Approval satisfied, pipeline ignored. The MR has the required approvals and merges even though the pipeline is still running or failed, because the target branch never had “Pipelines must succeed” enabled.
5. Fork or unprotected-branch MR merging without a gate note at all. No plan-gate note appears and the MR merges, because the pipeline ran without the protected token and the job was skipped or failed open.

If symptom 1 or 2 is present with a max_cost_multiplier above 3.0x, stop auto-merge on the target branch immediately and work the remediation below before re-enabling it.

Root Cause Analysis #

allow_failure on the gate job. When the plan-gate job carries allow_failure: true, GitLab records an exit-1 as a non-blocking failure and the pipeline reports passed. This is the most frequent cause of symptom 2. Confirm the compiled config for the job:

glab ci config compile \
  | python -c "import sys,yaml,json; c=yaml.safe_load(sys.stdin); print(json.dumps(c.get('plan-gate',{}).get('allow_failure','<unset>')))"

An output of true — or an allow_failure clause inside the rules: entry that matches merge-request pipelines — is the defect.

Merge-request approval rules versus pipeline status. Approvals and pipeline status are independent gates in GitLab. An MR can satisfy every approval rule and still merge with a failed pipeline unless the branch requires green pipelines. Check the effective merge-method and pipeline requirement:

glab api "projects/$CI_PROJECT_ID" \
  | jq -r '"only_allow_merge_if_pipeline_succeeds=\(.only_allow_merge_if_pipeline_succeeds)\nmerge_method=\(.merge_method)"'

If only_allow_merge_if_pipeline_succeeds is false, the pipeline result — including your gate — is advisory, not binding.

Detached versus merged-results pipelines. A detached MR pipeline runs against the source branch head; a merged-results pipeline runs against a simulated merge with the target. If the gate only runs on the detached pipeline, a regression introduced by the merge itself is never scored. Inspect which pipeline type gated the MR:

glab api "projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/pipelines" \
  | jq -r '.[0] | "\(.source)\t\(.status)\tref=\(.ref)"'

A source of merge_request_event with a refs/merge-requests/…/merge ref confirms merged-results; a …/head ref means only the detached pipeline ran.

Missing “Pipelines must succeed” setting. Even with a red pipeline and allow_failure: false, a branch without the project-level merge requirement will let the MR merge. This is the setting that binds pipeline status to mergeability, and it is off by default on many self-managed instances.

Step-by-Step Remediation #

1. Remove allow_failure from the gate on merge-request pipelines. Pin the plan-gate job’s rules: so the merge-request branch never allows the job to fail:

plan-gate:
  stage: plan-gate
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      allow_failure: false
    - when: never
  script:
    - python -m plan_gate.gitlab

The trailing when: never guarantees the job exists only on merge-request pipelines and never inherits a permissive default from a branch pipeline.

2. Require merged-results pipelines through workflow: rules. Force every MR to be gated by a pipeline that runs against the merge result:

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
    - when: never

3. Enable “Pipelines must succeed” on the project. Bind pipeline status to mergeability via the REST API:

glab api --method PUT "projects/$CI_PROJECT_ID" \
  --field only_allow_merge_if_pipeline_succeeds=true \
  --field only_allow_merge_if_all_discussions_are_resolved=true

Expected output includes the updated flags:

{
  "only_allow_merge_if_pipeline_succeeds": true,
  "only_allow_merge_if_all_discussions_are_resolved": true,
  "merge_method": "merge"
}

4. Make the gate resolve discussions so a FAIL is inescapable. Have the gate open a non-resolvable blocking discussion on FAIL rather than a plain note, so even a green-pipeline race cannot merge past it. The gate posts a discussion through the REST API:

import os
import httpx

async def open_blocking_discussion(multiplier: float) -> None:
    api = os.environ.get("CI_API_V4_URL", "https://gitlab.com/api/v4")
    project = os.environ["CI_PROJECT_ID"]
    mr = os.environ["CI_MERGE_REQUEST_IID"]
    body = (
        f"⛔ **Plan cost regression blocks this merge** — "
        f"max cost multiplier `{multiplier:.2f}x` exceeds the 3.0x gate. "
        f"Resolve the regression or override with an approved exception label."
    )
    async with httpx.AsyncClient(timeout=5.0) as client:
        resp = await client.post(
            f"{api}/projects/{project}/merge_requests/{mr}/discussions",
            headers={"PRIVATE-TOKEN": os.environ["CI_GITLAB_TOKEN"]},
            json={"body": body},
        )
        resp.raise_for_status()

With “All discussions must be resolved” enabled in step 3, the open discussion holds the merge even if a later pipeline flakes green.

5. Verify the block end to end with a synthetic regression. Push a branch whose verdict is seeded to FAIL, open an MR, and confirm the merge is refused:

glab mr create --source-branch plan-gate-selftest --target-branch main \
  --title "selftest: forced plan regression" --description "gate self-test" --yes
glab api "projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/merge" \
  --method PUT 2>&1 | jq -r '.message // .state'

Expected output confirms the gate held:

405 Method Not Allowed
"merge request cannot be merged: pipeline must succeed"

Verification Checklist #

• [ ] glab ci config compile shows allow_failure: false on the plan-gate job for merge-request pipelines.
• [ ] only_allow_merge_if_pipeline_succeeds is true on the project.
• [ ] only_allow_merge_if_all_discussions_are_resolved is true so a FAIL discussion holds the merge.
• [ ] The gating pipeline for a test MR reports source=merge_request_event with a merged-results ref.
• [ ] A seeded FAIL verdict with max_cost_multiplier above 3.0x leaves the MR unmergeable.
• [ ] A WARN verdict (multiplier between 1.5x and 3.0x) posts a note but does not block, if BLOCK_ALLOW_FAILURE=false.
• [ ] The verdict-store timeout path still exits 1, so a database outage blocks rather than passes.
• [ ] “Merge when pipeline succeeds” waits for the plan-gate stage, not just an earlier stage’s success.

Compatibility and Engine-Specific Notes #

The blocking mechanics differ between GitLab deployments; map them before relying on the gate.

On merge trains the gate must run inside the train pipeline, because a train merges the exact combined commit and an earlier detached verdict no longer describes what lands. On self-managed instances without a Premium license, merged-results pipelines are unavailable, so gate the detached MR pipeline and additionally gate the target-branch pipeline as a backstop. In all cases the cost thresholds themselves come from the upstream scoring in Tracking Cost Deltas Across Baseline Versions; this runbook only ensures the resulting FAIL band is honoured at merge time.

Related #

• Wiring Query Plan Regression Gates into GitHub Actions — the equivalent branch-protection model using required check runs.
• Enforcing Plan Baselines with Argo CD — blocking a GitOps sync on the same verdict contract.
• Tracking Cost Deltas Across Baseline Versions — where the max_cost_multiplier this runbook gates on is computed.
• CI/CD Gate Integration for Query Plan Regression — the broader area this runbook sits within.

← Back to Wiring Query Plan Regression Gates into GitLab CI