Versioning Baseline Metadata Schemas with JSON Schema

Versioning the JSON Schema that validates baseline metadata means evolving the contract so new captures gain fields while every historical artifact ever written still validates and still deserializes. This runbook documents the additive-versus-breaking change taxonomy, the $id and schemaVersion conventions that let one validator resolve many generations, the compatibility gate that runs in CI before a schema merge, and the versioned migration that rewrites stored envelopes in place without a read outage. It is the change-management discipline behind Schema Validation for Baseline Metadata and it produces the compatibility evidence consumed by Validating Schema Changes Against Baseline Metadata.

Baseline metadata is the envelope wrapped around each captured plan: the fingerprint, the engine and version, the capture timestamp, the cost vector, and the storage tier. That envelope is written once and read for the lifetime of the baseline. A careless schema edit — removing a field, tightening required, or flipping additionalProperties to false — does not fail on write; it fails months later when a validator rejects an artifact it wrote itself. The remedy is a versioned schema with an explicit compatibility policy enforced in CI.

Symptom Identification and Production Thresholds #

A schema regression is invisible until a read or a re-validation hits an old artifact. Treat each of the following as a hard breach condition with an exact trigger:

1. Post-bump validation failure spike. The validation-failure rate across stored artifacts exceeds 0.5% in the hour after a schema version is deployed, against a steady-state baseline under 0.02%. Any jump concentrated on artifacts older than the bump points at a breaking change, not bad data.
2. Rejected-artifact rate on write. New captures are rejected at more than 0.1% of write volume. Fresh writes failing the current schema mean the producer and the schema disagree — a coordination bug, not a legacy one.
3. Cross-version resolve miss. A validator cannot resolve the $id referenced by more than 0 stored artifacts. A single unresolvable $id means a retired schema generation was deleted while artifacts still reference it.
4. additionalProperties fallout. Setting additionalProperties: false rejects more than 50 distinct historical field names that were previously tolerated. This is the most common accidental breaking change and it fails silently against old envelopes.
5. Required-field addition backfill gap. A newly required field is absent from more than 1% of stored artifacts and no default or migration exists to supply it. Adding to required is breaking by definition for every artifact written before the field existed.
6. Schema-version skew. More than 3 distinct schemaVersion values are live in the store with no migration retiring the oldest. Unbounded generation sprawl makes the validator’s $ref resolution and the reader’s branching grow without limit.

When conditions 1 and 4 fire together, roll the schema $id back to the previous generation immediately; the change was breaking regardless of intent.

Root Cause Analysis #

Three failure domains turn a routine schema edit into a validation outage. Each has a direct diagnostic.

Breaking field removal or rename. Deleting a property that old artifacts still carry, or renaming it, breaks any reader that dereferences the old name and any schema branch that referenced it. Before merging, diff the property set the store actually uses against the proposed schema:

# Which property names appear in stored envelopes but are absent from the new schema?
jq -r '.properties | keys[]' schema/baseline.v2.json | sort > /tmp/schema_props.txt
jq -r '.metadata | keys[]' artifacts/*.json | sort -u > /tmp/stored_props.txt
comm -23 /tmp/stored_props.txt /tmp/schema_props.txt

Any line printed is a property present in stored data but missing from the new schema — a breaking removal unless the migration handles it.

additionalProperties: false on an evolving envelope. Locking the property set rejects every unknown field, so any forward-compatible field a newer producer wrote becomes a rejection under an older, stricter schema. Confirm the tolerance policy explicitly:

# Does the schema forbid unknown properties? (the silent breaker)
jq '.additionalProperties, .schemaVersion, .["$id"]' schema/baseline.v2.json

If additionalProperties is false on a metadata envelope that must accept forward-compatible fields, that is the regression; keep it true (or constrained via patternProperties) for the envelope’s open extension area.

Required-field addition without backfill. Adding a field to required invalidates every artifact written before that field existed. Quantify the gap against the real corpus before the gate can pass:

-- Fraction of stored artifacts missing the newly-required field.
SELECT round(100.0 * count(*) FILTER (WHERE metadata ? 'io_tier' IS NOT TRUE)
             / GREATEST(count(*), 1), 3) AS pct_missing_io_tier
FROM baseline_metadata;

A non-zero pct_missing_io_tier means the new required field needs a migration default before it can be enforced; otherwise every legacy row fails.

Step-by-Step Remediation #

1. Adopt an explicit version identity. Every schema carries a $id whose path encodes the major generation and a schemaVersion field that producers stamp onto each artifact. Additive changes bump the minor; breaking changes bump the major and mint a new $id path.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://queryplan.org/schema/baseline-metadata/v2",
  "title": "Baseline Metadata Envelope",
  "type": "object",
  "properties": {
    "schemaVersion": { "type": "string", "const": "2.3.0" },
    "fingerprint":   { "type": "string", "pattern": "^[0-9a-f]{64}$" },
    "engine":        { "type": "string", "enum": ["postgresql", "mysql", "cockroachdb"] },
    "capturedAt":    { "type": "string", "format": "date-time" },
    "costVector":    { "type": "array", "items": { "type": "number" }, "minItems": 1 },
    "ioTier":        { "type": "string", "enum": ["nvme", "ssd", "hdd"], "default": "ssd" }
  },
  "required": ["schemaVersion", "fingerprint", "engine", "capturedAt"],
  "additionalProperties": true
}

Note ioTier is not in required and additionalProperties stays true — that combination is what keeps the change additive.

2. Gate compatibility in CI. Before a schema merges, replay a corpus of stored artifacts of every retained generation against the proposed schema and classify the change. The gate is async, typed, and emits structlog events plus OpenTelemetry spans so a failed gate is traceable to the exact artifact and rule.

from __future__ import annotations

import asyncio
from dataclasses import dataclass, field
from enum import Enum

import structlog
from jsonschema import Draft202012Validator
from opentelemetry import trace

log = structlog.get_logger("schema.compat")
tracer = trace.get_tracer("schema.compat")

FAILURE_RATE_CEILING = 0.005     # 0.5% — above this the change is treated as breaking


class Compat(str, Enum):
    ADDITIVE = "additive"
    BREAKING = "breaking"


@dataclass
class CompatReport:
    schema_id: str
    checked: int = 0
    rejected: int = 0
    failing_ids: list[str] = field(default_factory=list)

    @property
    def failure_rate(self) -> float:
        return self.rejected / self.checked if self.checked else 0.0

    @property
    def verdict(self) -> Compat:
        return Compat.BREAKING if self.failure_rate > FAILURE_RATE_CEILING else Compat.ADDITIVE


async def check_compatibility(schema: dict, corpus: list[dict]) -> CompatReport:
    with tracer.start_as_current_span("check_compatibility") as span:
        span.set_attribute("schema.id", schema.get("$id", "unknown"))
        validator = Draft202012Validator(schema)
        report = CompatReport(schema_id=schema.get("$id", "unknown"))

        async def _one(artifact: dict) -> None:
            report.checked += 1
            errors = sorted(validator.iter_errors(artifact), key=lambda e: e.path)
            if errors:
                report.rejected += 1
                if len(report.failing_ids) < 25:
                    report.failing_ids.append(artifact.get("fingerprint", "?"))
                log.warning("artifact_rejected",
                            fingerprint=artifact.get("fingerprint"),
                            rule=errors[0].validator, path=list(errors[0].path))

        await asyncio.gather(*(_one(a) for a in corpus))
        span.set_attribute("compat.failure_rate", round(report.failure_rate, 5))
        span.set_attribute("compat.verdict", report.verdict.value)
        log.info("compat_report", schema_id=report.schema_id, checked=report.checked,
                 rejected=report.rejected, rate=round(report.failure_rate, 5),
                 verdict=report.verdict.value)
        if report.verdict is Compat.BREAKING:
            raise SystemExit(
                f"schema change is BREAKING: {report.rejected}/{report.checked} "
                f"legacy artifacts rejected ({report.failure_rate:.2%})")
        return report

3. Migrate stored envelopes for a breaking change. When the gate verdict is BREAKING and the change is intentional, mint the new major $id, then run a versioned migration that upgrades each stored envelope — supplying defaults for newly required fields and restamping schemaVersion — while the resolver keeps serving reads under both generations. Expected gate output on a clean additive change:

2026-07-18T11:20:04Z [info]    compat_report schema_id=https://queryplan.org/schema/baseline-metadata/v2 checked=48213 rejected=6 rate=0.00012 verdict=additive
2026-07-18T11:20:04Z [warning] artifact_rejected fingerprint=9c2f…41 rule=format path=['capturedAt']

The 6 rejects here are pre-existing malformed timestamps, well under the 0.5% ceiling, so the change deploys as a minor bump. The version resolver that maps each artifact’s $id to its validating schema is the same indirection the deterministic hashing described in Plan Hashing Algorithms for SQL Engines relies on to keep fingerprints stable across metadata generations.

Verification Checklist #

• [ ] Every stored artifact carries a schemaVersion and a $id that the version resolver can dereference.
• [ ] The CI compatibility gate replays the full retained corpus and reports a failure rate under 0.5% before merge.
• [ ] New captures are rejected at under 0.1% of write volume against the current schema.
• [ ] additionalProperties is true (or patternProperties-constrained) on the open extension area of the envelope.
• [ ] No newly required field is absent from any stored artifact after the migration backfill completes.
• [ ] At most 3 distinct schemaVersion values are live, and a migration is scheduled to retire the oldest.
• [ ] A round-trip test validates one artifact of every retained generation against the current validator set.
• [ ] The schema $id, schemaVersion, and compatibility verdict are recorded in the deploy audit log.

Compatibility and Engine-Specific Notes #

The versioning policy is portable; the practical constraints differ by validator and store.

The default keyword in Draft 2020-12 is an annotation, not an action — Python’s jsonschema will not inject it, so a newly required field must be backfilled by the migration UPDATE, not assumed. Ajv’s useDefaults does mutate the instance, which is convenient but means the validator and the writer must agree on ownership of the default. Whichever validator, keep every referenced $id resolvable for as long as any artifact carries it, and ground the enforcement mechanics in Validating Schema Changes Against Baseline Metadata.

Related #

• ← Back to Schema Validation for Baseline Metadata (parent topic)
• The enforcement gate this feeds: Validating Schema Changes Against Baseline Metadata
• Why fingerprints survive schema generations: Plan Hashing Algorithms for SQL Engines
• Wider context: Automated EXPLAIN Capture & Storage Workflows