Security Boundaries for Baseline Data Storage

The baseline persistence stage is the write-once, tamper-evident sink that accepts validated query-plan artifacts, enforces cryptographic and schema contracts at its edge, and hands an immutable corpus to the regression evaluators downstream. Establishing hard security boundaries around this store is what keeps historical execution metadata trustworthy: if an attacker or a buggy collector can rewrite yesterday’s baseline, every regression comparison built on top of it becomes meaningless. Within Core Architecture & Baselining Fundamentals, this stage sits after fingerprinting and normalization and before scoring, and it deliberately does nothing except validate, encrypt, route, and persist.

Architectural Boundaries: What This Stage Consumes and Emits #

The storage boundary consumes exactly one thing: a signed baseline payload emitted by the capture layer once a plan has been fingerprinted and normalized. It never reaches back into the database, never re-runs EXPLAIN, and never evaluates performance — those responsibilities belong to the capture layer and the regression engines respectively. Enforcing that isolation is what lets the store guarantee write-once-read-many (WORM) semantics.

Upstream, the contract is a canonical JSON envelope containing a plan identifier produced by the SHA-256 fingerprinting approach, a normalized cost vector, an engine discriminator, a capture timestamp, and a detached ECDSA signature. Cost fields arrive pre-normalized against the unified representation defined in Cost Estimation Mapping Across PostgreSQL and MySQL, so the persistence stage validates dimensions rather than re-deriving them. Structural metadata is expected to already satisfy the rules in Schema Validation for Baseline Metadata; the storage boundary treats that contract as a hard gate, not a suggestion.

Downstream, the stage emits two artifacts. The first is an immutable archive object — an encrypted, object-locked record keyed by plan identifier — that becomes the source of truth for every later comparison against defined regression thresholds. The second is a structured audit event on the rejection bus for any payload that fails a boundary check. Crucially, a rejected payload never touches the archive: validation is synchronous and fail-closed, so a malformed or unsigned record cannot silently poison the corpus that the regression detection engines will later trust.

Three enforcement points define the boundary. The ingestion gateway authenticates the caller and verifies the detached signature. The verification stage recomputes the plan hash and rejects any identifier mismatch caused by optimizer drift or a tampered payload. The persistence stage applies server-side encryption with customer-managed keys and an immutable retention policy before the object is durably committed. All three run inside a zero-trust data plane isolated by network segmentation from general telemetry traffic.

Deterministic Routing and Schema Enforcement #

Every payload is classified into exactly one route before any write occurs, and the routing decision is a pure function of the payload plus the caller’s verified identity — there is no ambiguity and no partial write. A payload is routed to the immutable archive only if it passes signature verification, hash re-computation, and schema validation in that order; a failure at any stage short-circuits to the matching dead-letter queue.

The field contract is expressed as a JSON Schema so that the same definition can gate both the ingestion service and the replay workers:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://queryplan.org/schemas/baseline-payload.v3.json",
  "type": "object",
  "required": ["plan_id", "engine", "cost_vector", "signature", "captured_at_ms"],
  "additionalProperties": false,
  "properties": {
    "plan_id":     { "type": "string", "pattern": "^[a-f0-9]{64}$" },
    "engine":      { "type": "string", "enum": ["postgresql", "mysql"] },
    "schema_ver":  { "type": "integer", "const": 3 },
    "captured_at_ms": { "type": "integer", "minimum": 1704067200000 },
    "signature":   { "type": "string", "pattern": "^[a-f0-9]{128,144}$" },
    "cost_vector": {
      "type": "object",
      "required": ["cpu_cost", "io_cost", "memory_cost", "row_estimate"],
      "additionalProperties": false,
      "properties": {
        "cpu_cost":     { "type": "number", "minimum": 0, "maximum": 1e12 },
        "io_cost":      { "type": "number", "minimum": 0, "maximum": 1e12 },
        "memory_cost":  { "type": "number", "minimum": 0, "maximum": 1e12 },
        "row_estimate": { "type": "integer", "minimum": 0, "maximum": 9e15 }
      }
    }
  }
}

The partition key for the archive is derived deterministically so that the same plan always lands in the same partition and idempotent overwrites are impossible. The key formula is partition = engine + "/" + plan_id[0:2] + "/" + plan_id, which spreads writes across 256 hash-prefix shards per engine while keeping the object address a pure function of the identifier. Because plan_id is the SHA-256 fingerprint, a re-submission of an identical plan resolves to the identical object key, and the object-lock policy rejects the second write rather than mutating the first — that is the WORM guarantee expressed as a storage-layer invariant.

Routing targets and their retry semantics are declared explicitly so that the boundary’s behavior is auditable configuration rather than buried code:

# routing_matrix.yaml
ingestion:
  validation_timeout_ms: 500
  max_payload_size_kb: 128
  routing:
    success:
      target: "s3://baseline-archive/immutable/"
      retention_policy: "COMPLIANCE_365D"
      object_lock: true
      idempotency_key: "plan_id"
      max_retries: 0
    hash_mismatch:
      target: "sqs://baseline-dlq/hash-failure"
      retry: 0
      alert_severity: "P2"
      max_queue_depth: 10000
    schema_violation:
      target: "sqs://baseline-dlq/schema-failure"
      retry: 0
      alert_severity: "P3"
      max_queue_depth: 25000
    signature_invalid:
      target: "sqs://baseline-dlq/auth-failure"
      retry: 0
      alert_severity: "P1"
      max_queue_depth: 5000

max_retries: 0 on the success path is deliberate: because writes are content-addressed and object-locked, a retried write is either a redundant no-op or a policy violation, so retries are pushed onto the caller and surfaced as explicit idempotency conflicts rather than hidden inside the storage client.

Production-Ready Implementation #

The reference boundary is an asyncio service that verifies the signature, recomputes the hash, validates against the schema contract, and commits to object-locked storage — all under a single OpenTelemetry span with structlog structured logging. It uses asyncpg to record an append-only ledger row alongside the object write so that the audit trail is queryable without scanning the archive. The happy path is shown end to end; every rejection raises a typed exception that the router maps to a dead-letter target.

# baseline_storage_boundary.py
import asyncio
import hashlib
import json
from dataclasses import dataclass

import asyncpg
import structlog
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
from jsonschema import Draft202012Validator
from opentelemetry import trace, metrics

log = structlog.get_logger("baseline.storage")
tracer = trace.get_tracer("baseline.storage")
meter = metrics.get_meter("baseline.storage")

ROUTE_COUNTER = meter.create_counter("baseline_ingest_routed_total")
COMMIT_LATENCY = meter.create_histogram(
    "baseline_commit_duration_ms", unit="ms"
)

with open("baseline-payload.v3.json", "rb") as fh:
    _VALIDATOR = Draft202012Validator(json.load(fh))


class BoundaryRejection(Exception):
    """Raised when a payload fails a boundary check. `route` names the DLQ."""

    def __init__(self, route: str, reason: str):
        self.route = route
        self.reason = reason
        super().__init__(f"{route}: {reason}")


@dataclass(frozen=True)
class Boundary:
    pool: asyncpg.Pool
    object_store: "ObjectLockClient"
    public_key_pem: bytes

    def _verify_signature(self, payload: dict) -> None:
        signable = {k: v for k, v in payload.items() if k != "signature"}
        canonical = json.dumps(
            signable, sort_keys=True, separators=(",", ":")
        ).encode()
        key = serialization.load_pem_public_key(self.public_key_pem)
        try:
            key.verify(
                bytes.fromhex(payload["signature"]),
                canonical,
                ec.ECDSA(hashes.SHA256()),
            )
        except (InvalidSignature, ValueError) as exc:
            raise BoundaryRejection("signature_invalid", str(exc)) from exc

    def _verify_hash(self, payload: dict) -> None:
        # Recompute the fingerprint over the normalized, signature-free body.
        body = {k: v for k, v in payload.items() if k != "signature"}
        canonical = json.dumps(
            body, sort_keys=True, separators=(",", ":")
        ).encode()
        computed = hashlib.sha256(canonical).hexdigest()
        if computed != payload["plan_id"]:
            raise BoundaryRejection(
                "hash_mismatch",
                f"declared={payload['plan_id']} computed={computed}",
            )

    def _validate_schema(self, payload: dict) -> None:
        errors = sorted(
            _VALIDATOR.iter_errors(payload), key=lambda e: e.path
        )
        if errors:
            raise BoundaryRejection(
                "schema_violation", errors[0].message
            )

    @staticmethod
    def _partition_key(payload: dict) -> str:
        pid = payload["plan_id"]
        return f"{payload['engine']}/{pid[:2]}/{pid}"

    async def ingest(self, payload: dict) -> str:
        with tracer.start_as_current_span("baseline.ingest") as span:
            span.set_attribute("baseline.engine", payload.get("engine", "?"))
            loop = asyncio.get_running_loop()
            start = loop.time()
            try:
                # Signature -> hash -> schema, fail-closed and ordered.
                self._verify_signature(payload)
                self._verify_hash(payload)
                self._validate_schema(payload)

                key = self._partition_key(payload)
                span.set_attribute("baseline.object_key", key)

                # Object-lock commit is idempotent-reject: a second write
                # to an existing locked key raises and never mutates data.
                await self.object_store.put_locked(
                    key=key,
                    body=json.dumps(payload).encode(),
                    retention_days=365,
                )
                async with self.pool.acquire() as conn:
                    await conn.execute(
                        """
                        INSERT INTO baseline_ledger
                            (plan_id, engine, object_key, captured_at_ms)
                        VALUES ($1, $2, $3, $4)
                        ON CONFLICT (plan_id) DO NOTHING
                        """,
                        payload["plan_id"],
                        payload["engine"],
                        key,
                        payload["captured_at_ms"],
                    )

                ROUTE_COUNTER.add(1, {"route": "success"})
                COMMIT_LATENCY.record((loop.time() - start) * 1000)
                log.info("baseline.archived", plan_id=payload["plan_id"], key=key)
                return key

            except BoundaryRejection as rej:
                span.record_exception(rej)
                ROUTE_COUNTER.add(1, {"route": rej.route})
                log.warning(
                    "baseline.rejected",
                    route=rej.route,
                    reason=rej.reason,
                    plan_id=payload.get("plan_id"),
                )
                await self.object_store.dead_letter(rej.route, payload, rej.reason)
                raise

The dead_letter call preserves the original payload, its rejection route, and the failure reason so that a quarantined record can be replayed byte-for-byte once the upstream defect is fixed. Because verification runs before any archive write, a poisoned payload consumes gateway CPU and a DLQ slot but never a byte of immutable storage.

Threshold Table and Alerting #

The boundary publishes exact SLOs so that on-call engineers can distinguish a slow-but-healthy store from a boundary that has begun silently dropping baselines. All values are enforced, not aspirational.

The auth-failure queue is the highest-severity signal because a sustained rise there means valid callers cannot write, or an attacker is probing the boundary with forged signatures. The corresponding Prometheus rules page immediately rather than after a long for window:

groups:
  - name: baseline_storage_boundary
    rules:
      - alert: BaselineSignatureRejectSpike
        expr: |
          rate(baseline_ingest_routed_total{route="signature_invalid"}[5m])
            / rate(baseline_ingest_routed_total[5m]) > 0.02
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "Signature rejection ratio > 2% — valid writers blocked or forgery probe"

      - alert: BaselineAuthDLQSaturated
        expr: baseline_dlq_depth{queue="auth-failure"} >= 5000
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "Auth-failure DLQ at capacity — circuit breaker imminent"

      - alert: BaselineKEKStale
        expr: baseline_kek_age_days > 90
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Customer-managed KEK past 90-day rotation window"

Failure Scenarios and Root Cause Analysis #

1. Hash mismatch from optimizer drift. Symptom: baseline_hash_mismatch_total climbs after a minor database version upgrade, with payloads landing in hash-failure. Root cause: an engine patch altered a cost-model field that leaked into the fingerprint, so the recomputed hash no longer matches the declared identifier. Diagnostics: pull a sample with aws sqs receive-message --queue-url $HASH_DLQ and diff the normalized body against a known-good baseline; confirm the divergent key. Mitigation: patch the volatile-field strip list in the capture layer’s normalization step, then replay the DLQ — do not relax the boundary check, because the check is doing its job.

2. Forged or stale signatures. Symptom: signature_invalid ratio spikes; auth-failure DLQ fills. Root cause: either a collector is signing with a rotated-out private key whose public counterpart is no longer trusted, or an untrusted principal is probing the gateway. Diagnostics: group DLQ entries by source principal (SELECT principal, count(*) FROM baseline_ledger_rejects WHERE route='signature_invalid' GROUP BY 1); a single legitimate collector points to key drift, a spread of principals points to an attack. Mitigation: for key drift, redistribute the current public key to verifiers and re-sign; for probing, tighten IAM ingress and rotate the signing key pair.

3. Object-lock write conflicts. Symptom: baseline_object_lock_conflict_total rises and callers see idempotency errors. Root cause: an upstream retry loop is re-submitting already-archived plans, or two collectors fingerprinted the same plan concurrently. Diagnostics: correlate the conflicting plan_id against the ledger captured_at_ms; a near-identical timestamp confirms a duplicate submission race. Mitigation: make the collector treat a lock conflict as success — the content-addressed object already holds the correct data — and add a dedupe cache keyed on plan_id upstream of the gateway.

4. DLQ saturation and backpressure. Symptom: any baseline_dlq_depth gauge reaches its max_queue_depth. Root cause: a systemic upstream defect (bad normalization deploy) is rejecting a large fraction of traffic faster than replay can drain it. Diagnostics: check the deploy timeline against the depth inflection; inspect a batch of rejections for a common error message. Mitigation: the circuit breaker trips at the configured depth and halts ingestion, applying controlled backpressure to the capture agents; roll back the upstream deploy, then drain and replay.

5. Cross-boundary key access denial. Symptom: commits fail with AccessDenied on kms:GenerateDataKey when baselines cross accounts or regions. Root cause: the storage service account lacks scoped KMS grants on the customer-managed key. Diagnostics: aws kms describe-key --key-id $KEK and audit the key policy for the service principal. Mitigation: add an explicitly scoped grant — the precise least-privilege pattern is detailed in Encrypting Baseline Query Plans at Rest and in Transit, which owns the envelope-encryption and rotation workflow this stage depends on.

Configuration Reference #

The boundary is tuned through a small, explicit set of knobs; defaults are chosen for a fleet ingesting on the order of thousands of baselines per minute.

Keep BASELINE_INGEST_WORKERS at or below the asyncpg pool max_size so that ledger writes never become the bottleneck that starves the object-store commit. When raising BASELINE_MAX_PAYLOAD_KB for engines that emit verbose plans, raise the schema-violation DLQ max_queue_depth in step, since larger payloads increase the blast radius of a bad normalization deploy.

Related #

• Plan Hashing Algorithms for SQL Engines — produces the deterministic identifier this stage verifies.
• Cost Estimation Mapping Across PostgreSQL and MySQL — defines the normalized cost vector validated at the boundary.
• Encrypting Baseline Query Plans at Rest and in Transit — the envelope-encryption and key-rotation runbook this stage depends on.
• Schema Validation for Baseline Metadata — the upstream contract the storage boundary treats as a hard gate.
• Defining Regression Thresholds for Query Plans — the downstream consumer that trusts this immutable corpus.

← Back to Core Architecture & Baselining Fundamentals