Encrypting Baseline Query Plans at Rest and in Transit

Baseline query plans are the source of truth every regression comparison keys on, so the cryptographic controls that wrap them — AES-256-GCM envelope encryption at rest and enforced TLS 1.3 in transit — are what keep that corpus trustworthy and available. This runbook documents the exact failure thresholds, async remediation code, and incident patterns for platform teams operating the encryption layer of the baseline data storage boundary, the write-once sink that sits between plan capture and the regression evaluators inside Core Architecture & Baselining Fundamentals.

Symptom identification and production thresholds #

Baseline encryption faults rarely surface as an explicit DECRYPTION_FAILED error. They present as silent degradation in CI regression gates and anomalous latency in the plan-diff engine. Treat any of the following breach conditions as an active incident:

1. Baseline fetch latency — p99 > 2.5s on plan-retrieval endpoints indicates TLS handshake retries or a saturated KMS decryption queue.
2. CI pipeline stalls — a regression job hanging at fetch_baseline_plan for > 120s signals a cipher-suite mismatch between the collector and object storage.
3. Plan-diff nullification — when the regression engine receives a 0-byte or corrupted_base64 payload, envelope decryption has failed silently from key-version drift.
4. TLS alert frequency — SSLV3_ALERT_HANDSHAKE_FAILURE or TLS1_ALERT_PROTOCOL_VERSION appearing at > 5/min in collector logs confirms an in-transit negotiation breakdown.
5. Key age — a KMS/CloudHSM key older than 90 days without automated rotation trips a compliance block and raises the probability of stale ciphertext.

These thresholds feed the same alerting fabric that governs regression thresholds for query plans; a crypto breach must fail the gate closed, never wave a deploy through on a missing baseline.

Root cause analysis #

Most encryption regressions trace to three named failure domains. Each carries a fast diagnostic you can run before touching remediation code. The underlying model is envelope encryption — a per-record Data Encryption Key (DEK) encrypts the plan JSON, and a KMS-managed Key Encryption Key (KEK) wraps the DEK.

1. Envelope key drift. If the KEK rotates but the DEK metadata (key_id, algorithm, iv) is not atomically updated in the baseline manifest, the regression engine unwraps with stale parameters. Confirm the manifest key_id still points at a live key:

aws kms describe-key --key-id "$BASELINE_KEK_ARN" \
  --query 'KeyMetadata.[KeyState,DeletionDate,KeyManager]' --output text
# Expected: Enabled  None  CUSTOMER

Refer to the AWS guide on envelope encryption for the canonical DEK/KEK workflow.

2. In-transit cipher downgrade. Legacy CI runners or old collectors default to weak cipher suites that modern storage rejects, resetting the connection before payload transfer. Verify the endpoint actually negotiates TLS 1.3:

openssl s_client -connect "$BASELINE_STORAGE_HOST":443 -tls1_3 </dev/null 2>/dev/null \
  | grep -E 'Protocol|Cipher'
# Expected: Protocol : TLSv1.3   Cipher : TLS_AES_256_GCM_SHA384

3. Storage boundary misconfiguration. When baseline artifacts cross availability zones or accounts, IAM/KMS policies often omit kms:Decrypt or kms:GenerateDataKey for the regression service account. This is the same least-privilege enforcement described in Security Boundaries for Baseline Data Storage, where cross-boundary key access must be explicitly scoped to prevent lateral decryption. Check the caller’s effective grant:

aws sts get-caller-identity --query 'Arn' --output text
aws kms list-grants --key-id "$BASELINE_KEK_ARN" \
  --query "Grants[?GranteePrincipal=='$REGRESSION_ROLE_ARN'].Operations"
# Expected to include: Decrypt, GenerateDataKey

Step-by-step remediation #

Remediation requires a deterministic envelope handler, strict TLS enforcement in the fetch path, and automated key-rotation validation. The model below encrypts each plan with a fresh DEK and wraps that DEK with the KMS-held KEK, which never leaves KMS.

1. Deploy the async envelope handler #

The handler is asyncio-native so it shares the event loop with the ingestion pipeline, and every operation emits a structured log line plus an OpenTelemetry span for the observability hooks below. AES-GCM supplies both confidentiality and an authentication tag, so a tampered ciphertext raises on decrypt rather than returning garbage.

import os
import base64
import structlog
import aioboto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from opentelemetry import trace

log = structlog.get_logger("baseline.crypto")
tracer = trace.get_tracer("baseline.crypto")

DEK_BYTES = 32   # AES-256
IV_BYTES = 12    # 96-bit GCM nonce; never reuse with the same DEK


class BaselinePlanCrypto:
    """Async envelope encryption for baseline plan artifacts (AES-256-GCM + KMS-wrapped DEK)."""

    def __init__(self, session: aioboto3.Session, key_id: str):
        self._session = session
        self._key_id = key_id

    async def encrypt_plan(self, plan_json: str) -> dict:
        with tracer.start_as_current_span("baseline.encrypt") as span:
            dek = os.urandom(DEK_BYTES)
            iv = os.urandom(IV_BYTES)
            ciphertext = AESGCM(dek).encrypt(iv, plan_json.encode("utf-8"), None)
            async with self._session.client("kms") as kms:
                wrapped = await kms.encrypt(KeyId=self._key_id, Plaintext=dek)
            span.set_attribute("kms.key_id", self._key_id)
            log.info("plan_encrypted", ct_bytes=len(ciphertext), key_id=self._key_id)
            return {
                "ciphertext": base64.b64encode(ciphertext).decode(),
                "wrapped_dek": base64.b64encode(wrapped["CiphertextBlob"]).decode(),
                "iv": base64.b64encode(iv).decode(),
                "key_id": self._key_id,
                "algorithm": "AES_256_GCM",
                "schema_ver": 3,
            }

    async def decrypt_plan(self, envelope: dict) -> str:
        with tracer.start_as_current_span("baseline.decrypt") as span:
            span.set_attribute("kms.key_id", envelope.get("key_id", "unknown"))
            wrapped = base64.b64decode(envelope["wrapped_dek"])
            async with self._session.client("kms") as kms:
                unwrap = await kms.decrypt(CiphertextBlob=wrapped)
            iv = base64.b64decode(envelope["iv"])
            ciphertext = base64.b64decode(envelope["ciphertext"])
            try:
                plaintext = AESGCM(unwrap["Plaintext"]).decrypt(iv, ciphertext, None)
            except Exception:
                log.error("envelope_auth_failure", key_id=envelope.get("key_id"))
                raise
            return plaintext.decode("utf-8")

The emitted envelope carries schema_ver, so a canonicalization or key-policy change produces a new logical namespace instead of silently mutating records already committed under the schema-validated baseline metadata contract.

2. Enforce TLS 1.3 and validate the envelope in the CI gate #

Embed cryptographic validation directly into the regression gate so a broken envelope blocks the diff stage rather than corrupting the comparison. This snippet pins TLS 1.3 on fetch, then asserts envelope structure before any decrypt runs.

stages:
  - fetch_baseline
  - validate_crypto
  - regression_diff

fetch_baseline:
  stage: fetch_baseline
  image: python:3.13-slim
  script:
    - pip install boto3 cryptography requests
    - python -c "
      import ssl, requests
      ctx = ssl.create_default_context()
      ctx.minimum_version = ssl.TLSVersion.TLSv1_3
      resp = requests.get('${BASELINE_STORAGE_URL}', verify=True, timeout=30)
      resp.raise_for_status()
      open('/tmp/plan_envelope.json', 'w').write(resp.text)
      "
  artifacts:
    paths: ["/tmp/plan_envelope.json"]
    expire_in: 1h

validate_crypto:
  stage: validate_crypto
  script:
    - python -c "
      import json, sys
      env = json.load(open('/tmp/plan_envelope.json'))
      required = {'ciphertext', 'wrapped_dek', 'iv', 'key_id', 'algorithm'}
      missing = required - env.keys()
      if missing:
          sys.exit(f'FATAL: missing envelope fields: {missing}')
      print('Envelope structure validated successfully.')
      "

Expected validate_crypto output on success: Envelope structure validated successfully.; on a truncated payload the job exits non-zero with FATAL: missing envelope fields: {'iv'} and the pipeline halts before the diff.

3. Wire observability so silent failures page early #

Cryptographic pipelines fail quietly, so instrument them explicitly. Emit these named metrics and alert on the exact thresholds.

groups:
  - name: baseline_crypto_alerts
    rules:
      - alert: BaselineDecryptionLatencyHigh
        expr: histogram_quantile(0.95, rate(baseline_kms_decrypt_duration_seconds_bucket[5m])) > 0.8
        for: 3m
        labels: { severity: warning }
        annotations:
          summary: "KMS decryption latency exceeding 800ms p95"
      - alert: BaselineEnvelopeCorruption
        expr: increase(baseline_envelope_decryption_errors_total[10m]) > 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "Baseline plan decryption failures detected — check KEK rotation status."

4. Define safe fallback chains for crypto-dependency failure #

When a cryptographic dependency fails, the pipeline must degrade without weakening posture:

1. Circuit breaker — if the kms:Decrypt error rate exceeds 10% over a 2-minute window, trip the breaker and route fetches to a read-only cache.
2. Stale-baseline fallback — serve the most recently validated plan from an encrypted, short-TTL cache (TTL=15m), tagged x-baseline-state: cached_fallback so it cannot raise a false-positive regression.
3. Break-glass key override — during a KMS regional outage, authorized SREs inject a pre-approved, hardware-backed recovery key via BASELINE_RECOVERY_KEY_ARN; the grant is audited and auto-revoked after 24h.
4. Fail-closed bypass — if decryption fails with no fallback, skip the diff stage but block the release with a CRYPTO_DEPENDENCY_DEGRADED status and page the platform on-call. Never auto-approve.

For rotation, do not use aws kms schedule-key-deletion (that destroys a key); create a new KEK with aws kms create-key and re-encrypt the affected envelopes.

Verification checklist #

Run each item after applying a fix; every box must be checked before clearing the incident.

• [ ] aws kms describe-key reports KeyState=Enabled and DeletionDate=None for the manifest key_id.
• [ ] openssl s_client -tls1_3 against the storage endpoint negotiates TLSv1.3 and TLS_AES_256_GCM_SHA384.
• [ ] A round-trip encrypt_plan → decrypt_plan returns byte-identical JSON for a sample baseline.
• [ ] baseline_envelope_decryption_errors_total holds at 0 for 15 minutes after redeploy.
• [ ] baseline_kms_decrypt_duration_seconds p95 is back under 0.8s.
• [ ] The regression service role lists both Decrypt and GenerateDataKey grants on the KEK.
• [ ] baseline_key_age_days for every active key_id is under 85.

Compatibility and engine-specific notes #

The envelope model is engine-agnostic because plans are stored as canonical JSON in object storage, but the collector’s in-transit configuration and the underlying at-rest option differ by source engine.

Whichever engine emits the plan, the cost fields arriving at this stage are already unified by Cost Estimation Mapping Across PostgreSQL and MySQL, and the plan identifier is the SHA-256 fingerprint the envelope is keyed on — so the encryption layer treats every engine’s artifact through one code path.

Related #

• Security Boundaries for Baseline Data Storage — the parent stage that authenticates, validates, and object-locks every payload this page encrypts.
• Schema Validation for Baseline Metadata — the upstream contract the envelope’s schema_ver field pins against.
• Defining Regression Thresholds for Query Plans — the downstream consumer that fails closed when a baseline cannot be decrypted.

← Back to Security Boundaries for Baseline Data Storage