Failing a Pull Request on Query Plan Regression in GitHub Actions

This runbook fixes the specific failure where a query plan regression is detected but the pull request still shows a green check and remains mergeable. It walks the exact symptoms — a stuck-green check despite a large cost delta, a missing annotation, an unblocked merge button — back to their four common root causes, then gives runnable YAML, Python, and gh CLI remediation with expected output. It is the enforcement companion to Wiring Query Plan Regression Gates into GitHub Actions, which covers the gate implementation itself.

The distinction that matters throughout: posting a failure conclusion and blocking merge on it are two independent facts. A gate can score correctly, post correctly, and still fail to stop a merge if the conclusion band is wrong, the job swallows its own exit code, the permissions are missing, or branch protection never required the check. Each of those is diagnosable in under a minute.

Symptom Identification and Production Thresholds #

Treat each of the following as a hard breach — an exact, observable condition, not a judgement call:

1. Check stays green despite a real regression. The scored verdict is FAIL with a cost_delta_ratio above $3.0\times$, yet the query-plan-regression check reports success. Any FAIL verdict that does not surface as a red check is a total enforcement failure.
2. Conclusion posted as neutral instead of failure. The Check Run exists and is grey, not red, for a FAIL verdict. neutral never blocks merge by default, so a FAIL mapped to neutral is functionally green.
3. Annotation missing on the changed SQL file. The check reports but no inline annotation appears in the “Files changed” tab, so a reviewer approving the PR sees no cost-delta context. Expected: exactly one annotation per changed query fingerprint.
4. Merge button enabled with a red check. The check is correctly red, but “Merge pull request” is still clickable for a non-admin. This means the check is advisory, not required — branch protection is not enforcing it.
5. Job reports success while the gate step failed. The workflow run is green even though the gate step logged a failure conclusion, because a downstream step or continue-on-error masked the non-zero exit. The job conclusion should equal the gate conclusion.

If symptom 1, 2, or 5 is present, a regression can merge silently; treat it as a release-blocking incident and page the platform on-call. Symptoms 3 and 4 are enforcement-integrity issues that degrade review quality but do not, alone, let a scored FAIL through.

Root Cause Analysis #

Branch protection misconfiguration. The check reports red but is not in the required list, so the merge button stays live. Confirm which checks are actually required on the target branch:

gh api /repos/$OWNER/$REPO/branches/main/protection/required_status_checks \
  --jq '{strict: .strict, contexts: [.checks[].context]}'

If query-plan-regression is absent from contexts, the check is advisory regardless of how red it looks.

neutral versus failure conclusion. The band mapping treated a FAIL as neutral, or REQUIRE_ACK downgraded it. Inspect the conclusion the gate actually posted for the head commit:

gh api /repos/$OWNER/$REPO/commits/$HEAD_SHA/check-runs \
  --jq '.check_runs[] | select(.name=="query-plan-regression") | {conclusion, external_id}'

A conclusion of neutral on a scored FAIL is the bug — neutral is reserved for WARN, and only failure blocks.

Job swallowing the exit code. A step has continue-on-error: true, or a later step overwrites the job result, so the workflow reports success even when the gate exits non-zero. Grep the workflow for the masking directive:

gh api /repos/$OWNER/$REPO/contents/.github/workflows/plan-gate.yml \
  --jq '.content' | base64 -d | grep -nE 'continue-on-error|exit 0|\|\| true'

Any hit on the gate job is a candidate: continue-on-error: true converts a failing step into a green job, and a trailing || true silences the exit code entirely.

Missing permissions: checks: write. The token cannot post a failure, so the gate falls back or no check appears at all. Confirm the workflow-level grant:

gh api /repos/$OWNER/$REPO/contents/.github/workflows/plan-gate.yml \
  --jq '.content' | base64 -d | grep -nA3 '^permissions:'

If checks: write is absent, the API returns 403 and, depending on your gate code, either no check is posted or a fail-closed path fires with a confusing message.

Step-by-Step Remediation #

1. Make the band mapping emit failure for FAIL. The verdict-to-conclusion projection must be a total function with failure for the blocking band. This is the authoritative snippet — it never downgrades a FAIL:

from typing import Literal

BAND_TO_CONCLUSION: dict[str, Literal["success", "neutral", "failure"]] = {
    "PASS": "success",
    "WARN": "neutral",
    "FAIL": "failure",
}

def resolve_conclusion(verdict: str, require_ack: bool) -> str:
    conclusion = BAND_TO_CONCLUSION[verdict]
    # REQUIRE_ACK may escalate WARN, but must NEVER downgrade FAIL.
    if verdict == "FAIL":
        return "failure"
    if verdict == "WARN" and require_ack:
        return "neutral"  # visible, still merge-blocked via required-ack re-run
    return conclusion

assert resolve_conclusion("FAIL", require_ack=True) == "failure"

The trailing assert is the regression guard: a FAIL under any REQUIRE_ACK setting must resolve to failure.

2. Remove the exit-code mask from the workflow. Delete continue-on-error from the gate job and let the step’s non-zero exit fail the job. The gate step should exit non-zero whenever it posts a failure:

jobs:
  query-plan-regression:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      checks: write
      statuses: write
    steps:
      - uses: actions/checkout@v4
      - name: Evaluate plan regression gate
        env:
          PLAN_GATE_DSN: ${{ secrets.PLAN_GATE_DSN }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          conclusion=$(python -m plan_gate.run \
            --repo "${{ github.repository }}" \
            --head-sha "${{ github.event.pull_request.head.sha }}")
          echo "gate conclusion: $conclusion"
          test "$conclusion" != "failure"

The final test line turns a failure conclusion into a non-zero exit, so the job itself goes red — belt and suspenders alongside the Check Run.

3. Add the check to branch protection. Register query-plan-regression as required and enable strict mode so the merge commit’s parents are always evaluated:

gh api -X PATCH /repos/$OWNER/$REPO/branches/main/protection/required_status_checks \
  -f strict=true \
  -f 'checks[][context]=query-plan-regression'

Expected output confirms the context is now required:

{
  "strict": true,
  "checks": [
    { "context": "query-plan-regression", "app_id": -1 }
  ]
}

4. Verify the check now blocks a real regression. Push a branch whose plan scores FAIL, open a PR, and read the mergeable state directly:

gh pr view $PR --json mergeable,mergeStateStatus \
  --jq '{mergeable, state: .mergeStateStatus}'

Expected output for a correctly blocked PR:

{
  "mergeable": "MERGEABLE",
  "state": "BLOCKED"
}

mergeStateStatus of BLOCKED with the red query-plan-regression check is the success condition: GitHub considers the branch clean but refuses the merge because a required check is failing.

Verification Checklist #

• [ ] resolve_conclusion("FAIL", require_ack=True) returns failure (assert passes in CI).
• [ ] The gate job has no continue-on-error: true and no || true on the gate step.
• [ ] The workflow declares permissions: checks: write and statuses: write at job or workflow level.
• [ ] gh api .../commits/$HEAD_SHA/check-runs shows conclusion: failure for a scored FAIL.
• [ ] Exactly one annotation appears on each changed SQL file in the “Files changed” tab.
• [ ] query-plan-regression is present in the branch’s required_status_checks.contexts.
• [ ] gh pr view $PR --json mergeStateStatus returns BLOCKED for a regressed PR.
• [ ] A non-admin cannot click “Merge pull request” while the check is red.

Compatibility and Engine-Specific Notes #

The enforcement mechanics differ across runner types, token types, and ruleset scope. Map your setup before applying the remediation above.

• GitHub App versus PAT. A GitHub App installation token scopes checks: write per repository and scales rate limits with the installation, which is why the parent guide recommends it over a personal access token for high-PR-volume orgs. A PAT works for a single repo but shares the user’s global rate budget and cannot post checks as a distinct bot identity.
• Org rulesets versus repo branch protection. An org-level ruleset can require query-plan-regression across every repo at once and cannot be overridden per repo, which is the durable way to prevent symptom 4 fleet-wide. Repo-level branch protection is faster to set up but drifts as new repos are created without it. Prefer an org ruleset once the gate is proven.
• Verdict scoring parity. The FAIL band and $3.0\times$ cost-delta threshold referenced here come from the detection stage, not the gate; if a plan scores FAIL inconsistently, the fix belongs in Tuning Thresholds for False Positive Reduction, not this runbook. For the same enforcement problem on a different platform, see Wiring Plan Regression Gates into GitLab CI, and for staged rollout of a newly-blocking gate see Designing Canary and Progressive Delivery Gates.

Related #

• Wiring Query Plan Regression Gates into GitHub Actions — the gate implementation, verdict contract, and SLOs this runbook enforces.
• Wiring Plan Regression Gates into GitLab CI — the same red-PR enforcement on GitLab pipelines.
• Designing Canary and Progressive Delivery Gates — rolling a blocking gate out without breaking every open PR at once.
• Tuning Thresholds for False Positive Reduction — where the FAIL band and cost-delta thresholds are actually set.
• CI/CD Gate Integration for Query Plan Regression — the wider gate-integration area.

← Back to Wiring Query Plan Regression Gates into GitHub Actions