The Cross-Stack AI Architecture Review Guide

Maggie Nanyonga · 2026-07-21 · AI Infrastructure, MLOps, Software Architecture, AI Governance, Engineering Management

Checklists, revalidation triggers, a release lifecycle, and system metrics for reviewing the seams of a production AI stack.

Templates, trigger classes, a release lifecycle, and a metric worksheet for reviewing the seams of a production AI stack

The argument this guide rests on is made once, in The AI Stack Is a System, and is not re-argued here.

What this document is instead: the instruments. A one-page summary of the map, three templates (an evidence package, an architecture-review checklist, a one-page incident review), a revalidation policy with its decision tree, an eight-stage release lifecycle with its own one-page checklist, a fifteen-metric worksheet, and a short checklist for testing your tests. The stance under every section is the same: review the seams, not the boxes. Your component reviews are probably fine. The twenty-two failure chains behind this material rarely contain an unreviewed component. They contain boundaries nobody owned and changes nobody classified. The warning signals usually existed; nobody was obliged to answer them. (The program's incident register numbers those chains CH01 through CH22; where I cite one, I name the incident and keep the number, so you can find the full writeup and primary sources in the flagship's appendix. "Incident register" always means that numbered set of chains; the contract register, which appears later, is a different object: the program's record of interface contracts and ownership entries.)

Two notational habits carry over from the flagship. Four coined terms get an asterisk at first use (domain, interface contract, feedback path, and the dual-membership ownership rule) because they carry weight and have no settled names; the flagship's glossary gives each its nearest established relative. And any rule or term that is my construction rather than a sourced claim is labeled [author-position]. Everything else traces to an operator postmortem, a regulatory filing, a specification, or a standards document.

1. The map, in one page

A production AI stack partitions into five functional domains*: a physical and compute foundation (accelerators, drivers, interconnect, byte and object storage); compute and platform orchestration (admission, placement, multi-tenancy); data infrastructure (tables, catalogs, feature pipelines, indexes: everything with data semantics); model and intelligence (training, model artifacts, serving engines and their internals, the model API surface); and context, retrieval, and applications (prompt assembly, retrieval pipelines, agent runtimes, the product). The partition rule is functional: a component belongs to the domain whose contract it fulfills, not the domain it resides in or resembles. vLLM's scheduler stays in the model domain because it schedules requests against an engine's internals; Airflow stays in data infrastructure because the contract it fulfills is pipeline execution.

Two functions refuse the partition and are modeled as planes crossing all five domains ("plane" is my construction [author-position]; its nearest established relative is the "cross-cutting function" of NIST's AI Risk Management Framework, extended here to a structure with owned state and hosted components). The evaluation and assurance plane runs lifecycle gates at boundaries plus continuous monitoring streams inside other domains' pipelines; it owns verdicts, baselines, and evidence, wherever the bytes physically live. The governance, security, and operations plane owns policy intent, audit state, and the enforcement points (admission controls, catalog interception, inline safety filters) that materialize policy inside the domains. Those points sit physically in the serving and data paths, which is why they can fail with the blast radius of infrastructure. The planes also split a role that layer thinking fuses: evaluation issues the verdict, from evidence it custodially owns; governance and operations execute the resulting action (block, promote, roll back, quarantine). Keep those separate and a verdict can later be invalidated without rewriting history about who decided what.

Every consequential boundary carries an interface contract*: what crosses, what is guaranteed about it, what evidence shows the guarantee holds, who is accountable when it fails, and what change forces re-verification. Fourteen contracts cover the map. Two clauses in each carry the planes into every seam: a gate clause (what must be re-verified before an artifact crosses) and a monitoring clause (what evidence must flow back after it does).

Production feeds back along five typed feedback paths*, each with its own producer, owner, timescale, and characteristic way of breaking: monitoring and drift telemetry to the evaluation plane; retraining data to data infrastructure; incident and audit evidence to the governance plane; evaluation verdicts back into plane-owned state; and user or business outcomes to the application domain. When a checklist below calls something "the outcome path doing the monitoring path's job," it means users detected what telemetry should have.

Ownership follows the dual-membership ownership rule*: for any component hosted in one domain while fulfilling another's contract (a guardrail in the serving path, a vector index in data infrastructure), the accountable owner is the domain whose contract it fulfills; the host owes hosting guarantees only; failure classification follows the breached clause.

System diagram of the amended five-domain AI-stack model: five domain boxes stacked vertically, from physical and compute foundation at the bottom to context, retrieval, and applications at the top, crossed by two vertical plane bands (governance, security and operations; evaluation and assurance) and fed by five typed production-feedback return paths, FP-1 to FP-5, drawn down the right side, with interface-contract marks on the seams between domains.

Visual 1: The amended five-domain systems model. Five functional domains, two cross-cutting planes, five typed production-feedback paths (FP-1 to FP-5), and interface contracts on the seams. Not every documented interface is drawn (14 contracts, C01 to C14). Source: Canonical Architecture v1.0, structural freeze 2026-07-08.

That is the whole apparatus. The rest of this guide is what you do with it.

2. Template: the evidence package

"Evidence package" is a coinage [author-position]. Every constituent exists in documented practice (audit trails, provenance and lineage records, model and system cards, registries), but no source I could verify names the unified, retained release record. This template is that record.

Custody, fixed in advance. The package is owned state of the evaluation plane across the artifact's whole lifecycle; the bytes live in data infrastructure; the governance plane consumes it as compliance evidence and executes verdict actions. The single failure mode this custody rule exists to prevent: an evidence write that silently fails and voids a "passed" gate. A pass without retained proof is not a pass.

Retention. Where the EU AI Act (Regulation 2024/1689) applies to a high-risk system, two cited floors exist: Article 12 requires the system to technically allow automatic recording of lifecycle events, and Article 26(6) requires deployers to keep those logs for no less than six months. Outside that scope the working rule is contractual: retain the record at least as long as any in-force verdict or deployed artifact pins it [author-position].

Package header (fill in): release or candidate ID; date and gate class (automated promotion filter, human-reviewed evaluation round, or release gate; all three are gates); verdict issuer and action executor, two names; storage location plus version pin (only atomically committed, named versions are pinnable); retention period and its basis.

The eleven items. For each, record the value or a resolvable pointer, who produced it, and an evidence status: complete, partial, or missing-with-reason. Test your pointers before you need them; a pointer that fails to resolve at audit time is the custody failure wearing a costume.

  1. Version manifest. The deployment-resolved list of every versioned artifact serving the release, recorded per instance: the deployed list, not the intended one. Knight Capital (CH19) is the canonical partial deploy: new code reached seven of eight servers, and the eighth executed a repurposed flag against retired code. The AI-stack form is Anthropic's 2025 routing bug (CH01), where the served configuration diverged from the advertised one. Include a served-vs-advertised configuration check.
  2. Dataset and source versions. Pinned dataset and snapshot versions with lineage pointers, for training and for evaluation. Unity's ad-targeting loss (CH13) reached the model because the data-to-model seam had no gated version boundary.
  3. Model and configuration versions. Base model, fine-tune, quantization, and serving configuration: server class, precision mode, sampling settings. All three Anthropic bugs were configuration or precision changes below the model API surface. Record what the gate actually ran against, in full.
  4. Tool and application versions. Prompt and template versions, tool schemas, agent-runtime versions. Verdicts bind to the full runtime triple: prompt, model, parameters. An assembly-side change can degrade outcomes with model versions unchanged (CH04, the context-cache incident).
  5. Evaluation versions. Benchmark-suite and scenario-set versions, baseline pins, judge and harness versions. A suite version change breaks cross-round comparability until it is re-established; living benchmarks drift by design.
  6. Policy versions. The versioned policy intent in force, per enforcement point, plus the enforcement-point inventory the release depends on. Google's Service Control outage (CH08) was a policy object with blank fields replicated globally; the Gemini safety-filter outage (CH12) was an enforcement point faulting inside the inference path. Both reconstruct only if this field exists.
  7. Test outcomes. Gate verdicts per lifecycle stage, each with a pointer to its custodial evidence, plus a record of which signals the gates measured. Outcomes can fail while evaluations pass; writing down what the gates looked at is what makes the later gate-versus-production comparison possible.
  8. Exceptions. Every waived, deferred, or overridden check; who granted it; the compensating control; the expiry date. An exception is a governance-plane action with recorded evidence, never a producing-team override. Cloudflare's regex outage (CH18) rode a rule-change path exempted from staged rollout; CrowdStrike's Channel File 291 (CH17) rode a content path exempt from the canary applied to code.
  9. Approval record. The named human or automated authority per gate, the evidence set in front of them at decision time, and the verdict-to-action handoff record. A signature without an evidence set is oversight theater.
  10. Canary evidence. Cohort, scope, duration, comparison against the gated baseline, and the promote-or-abort record. CrowdStrike's root cause turned on the absence of staged rollout for the artifact class that failed; the GPT-4o sycophancy release (CH03) shows the inverse gap: a behavioral regression that offline gates passed and only production observation caught.
  11. Rollback status. Whether rollback is possible, to what version, and what it does not undo. Cross-turn state, caches, indexes pinned to embedding versions, and policy compatibility do not roll back with a binary; declare them. Two of Anthropic's three 2025 bugs were resolved by rollback, which is exactly why rollback's preconditions belong in the package before release.

Completeness rule. A package missing any of items 1–7 is not a partial package; it is an ungated release with paperwork. Items 8–11 may legitimately be empty (no exceptions granted, no canary for this gate class, rollback not applicable), but the empty state is recorded as a decision [author-position].

3. Template: the architecture-review checklist

Run this against any proposed change to the map itself: domain boundaries, plane definitions, the feedback-path register, ownership assignments, or contract structure. Verdict per item: PASS, FAIL, or N-A with a reason. Any FAIL blocks the change. N-A without a reason counts as FAIL.

Section A: classify the change first

Section B: structural integrity

Section C: quality criteria (score the changed model, not the change)

Section D: ownership, security, and shared outcomes

Section E: labeling and evidence hygiene

Section F: revalidation propagation

Sign-off. Name, date, verdict (approve / approve with conditions / return), and the named decision this review improved. A review that cannot name one is ceremonial documentation, and the form should say so.

4. Template: the one-page incident review

Completed by the operating team the day after an incident, or as soon as mitigation stabilizes. Eleven fields. The one-page constraint is deliberate: a form an operator cannot complete the day after an incident will not be completed at all [author-position].

INCIDENT REVIEW: [ID], [date]

  1. Name. Precise and non-sensational: one line stating what failed and where.
  2. Initial condition. What changed, entered, or was already latent before the first symptom, with change and deploy IDs. In the Anthropic case, three independent changes were live by the time the investigation opened.
  3. Propagation path. Numbered steps. Every step names the domain or plane involved and the boundary crossed. If a step crosses no known contract, write "no contract": that is a finding, and the undeclared flow must be typed into the contract register before it keeps operating.
  4. Local signals. What each involved domain's own metrics showed at the time. Expect them to look healthy. That is the point of the field.
  5. Missing signal. The single highest-value field across the whole incident register: what measurement, had it existed, would have surfaced this failure earlier? Name it as a buildable metric. This field feeds the section-7 worksheet directly.
  6. System outcome. User- and business-horizon impact, quantified where possible. State availability and correctness separately, because they diverge.
  7. Detection evidence. What actually detected it (monitor, user reports, audit) and time to detection and mitigation. If users were the detector, say so plainly: that is the outcome path doing the monitoring path's job.
  8. Prevention vs mitigation. Two separate lists. Prevention stops recurrence, usually a gate or trigger change. Mitigation shortened this instance, usually detection or rollback. Do not let a good mitigation story substitute for prevention analysis.
  9. Ownership. For each breached clause: the accountable owner is the domain whose contract the failing component fulfills; the residence owner is named separately; classification follows the breached clause, wherever the process ran. A guardrail latency spike is a governance incident by the policy clause and a serving incident by the serving SLO; both routed, neither orphaned. If no accountable owner exists, record an ownership failure and treat it as the incident's most important output. Air Canada's chatbot ruling (CH15) is the canonical case of an ownership assignment eventually performed by a tribunal.
  10. Revalidation implication. Which triggers fired, or should have; which contract assumptions this incident invalidates; whether a new baseline is required. This is the field that turns an incident into governance evidence and updates the evaluation sets, routed three ways: incident record to audit state; missing signal to a monitoring clause; new scenario to a gate clause.
  11. Evidence status. Check one: observed incident / supported pattern across sources / derived scenario / open hypothesis. For an internal incident this is normally "observed." The field exists because reviews cite analogs and hypotheses, and those must not be laundered into observed fact.

Sources: postmortem links, telemetry snapshots, and the evidence-package pointers for the implicated release.

The three templates interlock. Field 10 of the incident review fires triggers whose outcomes land in the next release's evidence package (items 5–7). And repeated incident findings that implicate boundaries rather than components are the evidence stream that justifies invoking the architecture-review checklist on a boundary change.

5. When to retest: the seven trigger classes

A revalidation trigger [author-position] is a change event that invalidates prior assumptions and obliges a retest. The idea bridges to NIST's secure-development profile for generative AI (SP 800-218A), which recommends: "Retest AI models when they are retrained or new data sources are added." This policy is that one sentence generalized across the stack's fourteen seams [author-position].

Five rules frame everything below.

  1. Triggers fire on seam-semantics change only. That means a change to what a boundary promises: commit and visibility semantics, preemption notice, cache lifecycle, enforcement behavior, schema meaning, version identity. Never on a vendor, operator, or staffing change that preserves declared semantics. Without that scoping, the policy fires permanently and gets ignored within a quarter.
  2. The accountable owner of each affected interface executes the retest; the evaluation plane issues the verdict. No self-grading, especially when the party that changed the thing is the party being retested.
  3. The changing party declares the change as a gated, versioned event. Discovery of an undeclared change is itself a trigger (class 6).
  4. The cause and the required response are separate axes. Response vocabulary: no revalidation / targeted interface test / full regression / new baseline / architecture review / security review / human approval. The mapping from class to default response is my construction [author-position]: a pattern you should expect to override sometimes, deliberately and on the record.
  5. Every fired trigger produces retained evidence. A trigger that fires without a recorded outcome is indistinguishable from one that never fired.

| # | Trigger class | Core question | Default response | Grounding |

|---|---|---|---|---|

| 1 | Model / versioned-artifact change | Did a versioned artifact on the serving path (model, embedding model, prompt or template, tool schema) change version? | Full regression at the evaluation seams; targeted test at the context-assembly seam; acceptance gate before any index cutover | Anthropic's precision-path bug (CH01); the stale-index chain (CH22, a derived scenario; no public incident has yet been observed at that seam) |

| 2 | Data / schema change | Did data semantics change at a seam: schema, entity keys, snapshot or commit semantics, corpus scope, retention? | Targeted interface test; escalate to full regression when a training gate is reached; new baseline when pins are invalidated | the training-serving skew chain (CH21); Unity's contaminated corpus (CH13) |

| 3 | Config / routing change | Did configuration, routing, admission, caching, or automation velocity change while every component API stayed intact? | Targeted cross-seam conformance test (served vs advertised), never a unit test alone; escalate when harness semantics move or a sanctioned control path is deprecated | Anthropic's routing bug (CH01); the AWS DNS automation race (CH07); Cloudflare and Knight as out-of-AI analogues |

| 4 | Infrastructure relocation | Did a function, authority, or boundary move between domains, altitudes, or operators? | Architecture review first, then re-drawn targeted tests; security review stacks on if the privileged path or ACL surface moved; re-run the five-field ownership block | open risks flagged in the contracts themselves (the program calls these contract watch items), with no observed incident behind them [author-position] |

| 5 | Enforcement-point change | Did a governance enforcement point change, fail, or move class, including ACL models, attestation, or audit schemas? | Security review plus targeted test of the point itself; auto-invalidate dependent verdicts and regress the invalidated set; human approval where approval semantics changed | the Gemini safety-filter outage (CH12); Google Service Control (CH08) |

| 6 | Contract amendment | Did the measure or the contract register itself change (baselines, drift statistics, task-success definitions, ownership entries), or was an undeclared production flow discovered? | New baseline; architecture review for ownership changes; an undeclared flow is suspended until typed into the contract register | Zillow (CH14) and Tay (CH16): loop definitions never amended while the loop's behavior changed under them |

| 7 | External regulatory change | Did an applicable legal or framework obligation change? | Architecture review of affected seams plus targeted retention, logging, and locality tests; cite the instrument or the class does not fire | binding legal instruments [author-position] |

Three worksheet notes. One event can fire multiple classes: the Anthropic incident was class 3 (routing) and class 1 (precision path) at once; classify each constituent change and take the union of responses. Two cases deliberately dual-route: NIST's retrain-or-new-data case ("retrained or new data sources are added") is class 1 when the artifact changes and class 2 when the corpus scope changes; and a vendor absorbing maintenance is class 4 when authority over commits relocates, even though it superficially resembles an exempt vendor swap. Classes 4 and 7 rest on contract-flagged open risks and on binding legal instruments, with no observed incident behind either.

The explicit non-triggers

The scoping rule's value is as much in what does not fire. Each of the following is exempt, provided declared seam semantics are preserved:

  1. Operator or vendor swap preserving declared semantics. Swapping vector-database products fires nothing if consistency, freshness, and result-set semantics are unchanged.
  2. Personnel, team, or on-call rotation changes. Ownership attaches to domain contracts; individuals rotate. If a reorg breaks a named escalation interface, that is a class-6 contract-register amendment, and it does fire.
  3. Scaling events inside declared envelopes. Autoscaling, replica changes, and load variation within declared quota and admission envelopes are contract performance, not contract change.
  4. Residence relocation under identical residence terms. Moving hosted bytes between hosts owing the same guarantees fires nothing. Contrast class 4: it fires when authority moves, not bytes.
  5. Version bumps that cross no declared semantics boundary. Patch-level dependencies, engine upgrades, and format migrations that preserve commit and snapshot semantics.
  6. Cross-seam misrouting exclusions. An embedding-version change fires the index seam, explicitly not the context-assembly seam, unless the generation model also changed; a prompt promotion is gated per crossing, with no blanket pass.
  7. Observability additions. New telemetry, dashboards, or declared consumers of existing telemetry change no promise. Changing emission semantics does fire, and an undeclared consumer is the class-6 discovery case.

The doubt rule. The non-trigger list is an allowlist, not a presumption you argue from. A change that cannot be shown to preserve declared semantics classifies as a trigger. The asymmetry is evidence-based: the incident register holds multiple incidents caused by wrongly exempted changes and none caused by over-firing [author-position].

The decision-tree worksheet

Start at Q0 with the change event; exit with a response set.

CHANGE EVENT (declared by the changing party, or discovered,
which is itself class 6)
│
├─ Q0. Does the change alter what any seam PROMISES?
│      (commit/visibility semantics, preemption/drain notice,
│      cache lifecycle, enforcement behavior, schema meaning,
│      version identity of a pinned artifact, ownership or
│      authority)
│      Test: could a counterparty relying only on the contract
│      text observe different behavior?
│
│   NO ──► NON-TRIGGER. Response: none. Record the change in
│          the version manifest anyway. If in doubt, treat
│          doubt as YES: the cost of a wrongly exempted change
│          is silent staleness.
│   YES
│   ▼
├─ Q1. WHAT changed? Classify into class 1–7 (dual-route the
│      two spanning cases). Take the class's default response,
│      apply its escalations, and if multiple classes fire,
│      take the union of responses.
│
├─ Q2. For each affected interface: WHO executes?
│      The accountable owner of that interface retests its own
│      clauses. Residence owners deliver hosting evidence only.
│      The evaluation plane issues every verdict; no
│      self-grading. Disputes route by breached clause and end
│      at architecture-review authority.
│
└─ Q3. CLOSE-OUT: is the evidence retained?
       Response records are written into the release evidence
       package; verdicts are re-pinned to versions that will
       outlive them; the trigger, its classification, and its
       outcome are themselves logged.
       NOT CLOSED until the answer is yes. A fired trigger
       without retained evidence reproduces the voided-gate
       failure.

The tree deliberately has no expedited branch. Urgency is handled inside the response classes (the targeted interface test is the fast path), never by skipping Q0. Two of the incident register's chains are cases where the changing party would have self-classified as exempt.

6. The release-assurance lifecycle

"Release assurance" [author-position] names the lifecycle of checks that gates a change: cheap deterministic checks before expensive semantic evaluation, both before production exposure. Authority is fixed up front: the evaluation plane issues verdicts from evidence it custodially owns; the governance and operations plane executes the action; the producing team owns the artifact. The verdict-to-action handoff carries a declared latency SLO, because a gate whose verdicts arrive after the release window is a decoration.

Stage 1: Classify. Decision improved: which assurance path this change takes, and whether any artifact class may skip stages. Classify every candidate change against the trigger classes before anything runs. The grounding is blunt: the exemption is the failure. Cloudflare's WAF rule change bypassed staged rollout by standing procedure. CrowdStrike's Rapid Response Content went straight to global deployment as an entire artifact class exempted from the canary path. Knight repurposed an activation flag without asking what retired code paths still keyed on it. Any artifact class kept on a fast, canary-exempt path is itself a standing trigger whose exemption must be periodically re-justified.

Stage 2: Pre-merge deterministic validation. Decision improved: whether the change is structurally safe to build at all. Static analysis, error-handling review, feature-flag protection for new code paths in critical binaries, schema and type checks, tests on the changed path. Google's Service Control crash had neither error handling nor a feature flag on the failing path; a flag would have caught it in staging and served as a kill switch in production. And the validator is itself code: CrowdStrike's Content Validator passed a bad template because the validator had a bug. The bound: a clean stage 2 says nothing about behavior. Deterministic validation orders the pipeline. It does not conclude it.

Stage 3: Artifact-schema parity. Decision improved: whether the artifact the producer built is the artifact the consumer will execute. Assert producer-consumer schema parity at the exact seam the artifact crosses, and run the artifact inside the consuming environment rather than a simulation of it. CrowdStrike is the canonical case: the sensor expected 20 input fields, the update provided 21, and nothing validated the count in the environment that would execute it. For AI artifacts the check generalizes: model artifact against serving-engine version; prompt version against the model and parameters it was gated with; query-side embedding version against index build version; training-time against serving-time feature computation.

Stage 4: Scale-representative staging. Decision improved: whether pre-production evidence is valid at production scale, or merely at toy scale. Staging counts as evidence only if it is representative on the dimension the change actually stresses: cluster size, data volume, policy state, traffic shape. OpenAI's December 2024 telemetry deploy (CH06) failed only above a cluster-size threshold; a small staging cluster structurally could not surface it. Measure the load the change imposes on shared control planes as well as its own resource use. And state-representativeness counts as much as machine count: the Service Control path failed on a policy state that staging never held [author-position, generalizing the two incidents]. Whatever staging did not cover is declared, and compensated at canary.

Stage 5: Canary with class-specific signals. Decision improved: whether to expand exposure, decided on signals chosen for this change's plausible failure class, with pre-declared abort thresholds. Three ways to get this wrong, all observed. No canary at all: CrowdStrike. A canary blind to the artifact's binding non-functional property: Cloudflare needed per-rule worst-case CPU under adversarial input, which no functional test covers. And the deepest case, GPT-4o sycophancy: offline evals looked fine, the A/B numbers were positive, and the model still shipped a behavioral regression, because no deployment eval tracked the behavior that moved, and the acceptance signal (thumbs-up) was the same signal wired into the training reward, a self-confirming loop. Two rules follow. Canary signal sets include the behavior classes the change could plausibly move, as launch-blocking measures. And a signal that trains the model may not double as its acceptance metric [author-position]. One more, on authority: qualitative expert flags that the model feels off get a named decision owner with power to block, and overriding such a flag is a recorded governance decision, because in the sycophancy case the flag existed and the launch call went the other way.

Stage 6: Staged rollout with blast-radius bounds. Decision improved: how much damage the worst case can do while confidence is still partial. Bound the increments and the propagation velocity itself: Google's blank policy object replicated globally within seconds, which deleted the blast-radius boundary regardless of anyone's intent. Verify fleet version parity after each increment; Knight's eighth server is what that check exists for. Compute bounds against the dependency map: a one-percent config change to a component that fronts every model is a hundred-percent exposure, which is the Gemini filter in one sentence [author-position formulation; the common-mode fact is the source's]. Where the artifact emits actions, keep output-vs-input reconciliation live. And plan recovery load: restart herds are part of the radius.

Stage 7: Post-release monitoring window. Decision improved: when a release is actually done, and whether rollback fires in time. The release stays open, with heightened signals and pre-armed rollback, for a declared window after full exposure. The window inherits stage 5's class-specific signals, including user-horizon quality, because availability metrics cannot see a correctness collapse; the sycophancy regression was caught here, by usage and feedback, within about two days. Every window signal names a responder: Knight's systems emitted 97 reject emails before market open, and nobody was obliged to read them. The rollback path must not depend on the thing being rolled back: Cloudflare's rollback required running the failed build twice, and its own control-panel access ran through the failed product. The window closes by explicit verdict [author-position]; a timeout closes nothing except the paperwork.

Stage 8: Evidence archive. Decision improved: whether the next incident, audit, or revalidation can reconstruct what was released, why it passed, and what its verdict depended on. On window closure, the evidence package is sealed into evaluation-plane custody, including each verdict's enforcement-point dependency declarations. Those declarations are what make later invalidation possible: when the Gemini filter failed, verdicts that had assumed it healthy needed flagging, and that only works if the dependencies were written down while everyone was calm.

What counts as a release. The most repeated lesson across the incident register: the release process was applied to "software" while the incident arrived through something the organization did not treat as a release. In six of the chains grounding this lifecycle, the initiating artifact was a configuration or content change. The release-governed classes, from the evidence: model versions, including fine-tunes and quantization; prompts, templates, tool schemas, and agent graphs; configuration and flags, including flag repurposing and guardrail removal; enforcement points themselves (the gate that gates everything else does not get to ship ungated); data pipelines, datasets, indexes, and embedding versions; and observability instrumentation, whose emission cost is real load (the OpenAI telemetry deploy was the outage). Boundary statement [author-position]: model release does not equal application release. The sycophancy incident passed model-level gates and failed at the behavior horizon; the Gemini incident involved no model change at all and degraded every model's serving. The unit of release is the artifact plus its crossing.

The one-page release checklist

S1: Classify

S2: Deterministic

S3: Parity

S4: Staging

S5: Canary

S6: Rollout

S7: Window

S8: Evidence

Limits. Velocity versus assurance is a real trade, and no checklist defines it away: Cloudflare kept its emergency global path after the outage, on purpose, and the workable pattern is to keep emergency paths rare, give them their own compressed checklist, and re-justify them at stage 1 as standing exemptions [author-position]. Where full staging is impossible, bound velocity instead: limit how much an automated control can change per unit time, which is what AWS did after its health-check automation withdrew healthy capacity (CH07). Canaries catch late and cost real effort; the ordering exists because stages 2 through 4 catch earlier and cheaper. Some regressions are invisible at canary scale and window length; a clean stage 5 through 7 lowers risk without bounding it. And self-issued verdicts remain a structural ambiguity: CrowdStrike's validator and OpenAI's evaluations were both operated by the organization whose artifact they gated. This lifecycle does not resolve that; it only names it as open.

7. Measuring the seams: the metric worksheet

The method here inverts the usual direction. Rather than starting from what is easy to collect, every metric below was promoted because at least one incident chain's "missing signal" field shows it would have caught a real failure earlier. Each row names its owner (who produces the signal, who consumes it) and the decision it enables, because a metric with no decision attached is furniture. Four of the fifteen are my constructions and keep the label.

| ID | Metric | What it asks | Produced → consumed | Decision it enables |

|---|---|---|---|---|

| SM-01 | Evaluation-to-production correlation | Does the verdict that authorized this version predict what users actually get, per version, per slice? | model serving + application outcomes → evaluation plane | whether to trust the gate; persistent divergence means the gate evaluates the wrong thing (Anthropic, GPT-4o sycophancy, Zillow-adjacent) |

| SM-02 | Reliability by workload slice | Does success hold on the slices the evaluation envelope did not cover: server class, session length, cluster size, tenant, region? | application + model + orchestration labels → evaluation plane | which slice to freeze or drain; where to extend the evaluation envelope (Anthropic routing; OpenAI cluster-size threshold) |

| SM-03 [author-position] | Serving-seam output-quality monitor, sliced by hardware and config class | Is the semantic quality of output (coherence, unexpected-character rate, syntactic validity) uniform across the serving configurations actually used? | model output + infrastructure config labels → evaluation plane | which serving class to pull from rotation before users report it (Anthropic TPU bug; the 2024 gibberish-output incident, CH02) |

| SM-04 | Per-feature training-serving skew | Does the live serving distribution match the training distribution, per feature, at the data-model seam? | data pipelines + serving distributions → evaluation plane | whether to block the next training gate; which source to quarantine (skew chain CH21; Unity) |

| SM-05 | Grounded-answer rate + stale-answer rate | What fraction of served answers is verifiably supported by the retrieved evidence and consistent with the policy it cites? | application answer boundary + data freshness → evaluation and governance planes | when to force reindexing; whether the answer path can be trusted against its own sources (Air Canada; the stale-index scenario) |

| SM-06 | Golden-set recall per pinned version + version parity | Does retrieval quality hold per pinned embedding version, and does the query-side version equal the index build version? | data (index) + application (query side) + model (version announcements) → evaluation plane | accept or reject an index cutover (the stale-index scenario, CH22, derived and not yet observed; CrowdStrike's schema form) |

| SM-07 [author-position] | Cost per successful outcome + forecast reconciliation | What does a completed, verified task cost, and does the realized outcome match the model's own forecast, per cohort, at the outcome horizon? | application outcomes + model forecasts → application owns the mapping; evaluation plane consumes the delta | throttle or expand an automated operation before the quarterly filing does it for you (Zillow's $304M write-down; Unity) |

| SM-08 | Task completion at the horizon + loop continuity | Do multi-turn tasks complete over the full horizon, and does cross-turn state survive: cache hits versus clears, per turn? | application traces + model cache accounting → evaluation plane | gate agent releases at the trajectory horizon, where single-turn gates go blind (the context-cache incident, CH04) |

| SM-09 | Policy-compliant output rate + loop-contamination check | Are outputs compliant at emission, and is production data (adversarial or self-generated) shifting the learning distribution right now? | application + data ingestion + governance verdicts → governance and evaluation planes | halt a learning loop before it retrains on its own exhaust (Tay; Zillow) |

| SM-10 [author-position] | Automation velocity / blast radius / retry amplification | How fast is automation changing healthy capacity, how many independent failure domains does one control action touch, and how much does retry traffic amplify? | orchestration + foundation control loops → governance plane | set and enforce rate limits on automated control actions (five cloud postmortems, CH05/07/08/10/11; the strongest-grounded family in the set) |

| SM-11 | Enforcement-point liveness + common-mode map | Is every safety gate itself alive and measured, and how many services share a single enforcement dependency? | enforcement points fleet-wide → governance plane | strictest rollout bounds for common-mode components; alert on gate silence (Gemini filter; Meta's non-functional audit tool, CH09) |

| SM-12 [author-position] | Advertised-vs-usable ratio | How much of the claimed capacity, configuration, or reachability is actually usable end to end, measured off the failing substrate? | foundation + orchestration, with model and application config labels → cross-stack | detect the wrong-healthy-servers failure; check HA claims against actual placement (Anthropic routing; AWS 2021; Cloudflare facility, CH11) |

| SM-13 | Producer-consumer schema parity + canary crash and cost rate | Does what the producer validated equal what the consumer executes, and what does a small cohort show before global exposure? | release pipeline at the producing seam → evaluation plane gate | block promotion at the seam; abort at canary (CrowdStrike's 21-vs-20 fields; Cloudflare's regex CPU) |

| SM-14 | Cross-boundary conservation and attribution | Do outputs reconcile with inputs: orders out versus in, responses matched to requester identity, net exposure versus threshold? | application boundary + read paths + governance thresholds → cross-stack + governance plane | automated kill when the aggregate outcome breaches, with a named real-time owner (Knight's $460M; the 2023 cache identity leak, CH20) |

| SM-15 | Mean time to cross-layer root cause / to safe rollback | From the first outcome-level symptom, how long to identify a boundary-crossing root cause and complete a safe rollback? | all domains → governance plane, dependency-aware | whether your incident process can see boundary-crossing failures at all; where correlation is failing (AWS 2021's two lost hours; Gemini's disguised common origin) |

A disambiguation that has already bitten once. SM-15 is inspired by, but is not, the DORA "four keys." DORA here means DevOps Research and Assessment, the Accelerate research program. It does not mean the EU's Digital Operational Resilience Act, a distinct regulatory obligation that lives on the governance plane. The two senses genuinely co-occur in this territory; the conflation slipped into an earlier draft of this material and a red-team reviewer caught it. Spell out which DORA you mean, every time. The four keys measure delivery-team throughput per service; SM-15 measures system outcomes scoped to boundary-crossing failures, precisely the class a component-scoped "time to restore" misses.

Gaming note, applying to the whole table. The sharpest gaming risk observed live is the sycophancy case: a thumbs-up signal that both trained the model and served as its acceptance metric self-confirms. A signal used as a training reward may not simultaneously be the neutral acceptance metric for the same behavior. Second risk: easy proxies that invert under the failure actively conceal it. Zillow's conversion rate improved as the operation overpaid, because sellers accept above-market offers eagerly. SM-07 reconciles against the model's own forecast for exactly that reason.

The seven observability concerns

These are observability concerns, not new architectural boxes. Six map one-to-one onto the domains and planes; the seventh has no box of its own, on purpose.

| Observability concern | Where it lives on the map | Metrics it carries |

|---|---|---|

| Infrastructure (hardware, network, storage, scheduling, runtime) | foundation + orchestration | SM-10, SM-12 |

| Data (freshness, completeness, drift, schema, lineage, index state) | data infrastructure | SM-04, SM-05, SM-06 |

| Model (output distributions, latency, routing, token behavior) | model and intelligence | SM-03 |

| Application (traces, tool calls, workflow states, user outcomes, retry loops) | context, retrieval, and applications | SM-07, SM-08, SM-14 |

| Evaluation (dataset and judge versions, slice results, comparison validity) | evaluation plane | SM-01, SM-02, SM-06 (gate side) |

| Governance (identity, authorization, policy decisions, exceptions, audit completeness) | governance plane | SM-09, SM-11, SM-14 (identity) |

| Cross-stack (correlated traces, shared identifiers, version manifests, end-to-end outcomes, dependency-aware incident analysis) | the five typed feedback paths; no new domain | SM-01, SM-10, SM-11, SM-15 |

The load-bearing claim sits in the last row. Cross-stack observability is not a plane you build once; it is the property that emerges when every domain emits under its feedback-path contract and those emissions carry shared identifiers, version manifests, and aligned timestamps. The recurring pattern across the incident register is a chain failing at the cross-stack row while the component-local rows it crossed still looked healthy.

The evidence-custody rule

Observability data that a plane depends on is plane-owned from the moment of emission: physically resident in the host domain, custodially owned by the consuming plane, which owns baselines, comparison verdicts, and evidence packages. The host owes hosting SLOs and a named escalation interface. Three teeth:

  1. Custody from emission, no filtering en route. A "passed" gate whose evidence write silently failed is a voided verdict. SM-11's attestation reconciliation exists to catch this.
  2. A dead monitor mimics a healthy system. Silence is itself an alarm, which is why every emitter carries a runtime-health SLO, and why instrumentation gaps are a named failure class rather than an operational shrug.
  3. The plane may not be denied access to its own resident state. During an incident, the evaluation and governance planes must still reach their evidence even when the host domain is degraded. Which brings us to the anti-patterns.

Two anti-patterns, both seen live

Anti-pattern A: telemetry that shares fate with what it observes. In the 2021 AWS incident (CH05), the one metric that would have localized the root cause rode the same real-time monitoring pipeline the congestion took offline; operators fell back to logs and chased an internal-DNS symptom for about two hours. Rule: the signals that localize a substrate failure must be produced and reachable off that substrate. The subtler form is duplicated telemetry: two emitters for the same signal that share a hidden common dependency, redundancy in appearance only.

Anti-pattern B: timestamp misalignment across records. Cross-stack incident analysis works only if records across domains share a time base and correlation identifiers. In OpenAI's December 2024 outage, per-node DNS caching delayed visible failures, so the leading indicator surfaced at different wall-clock times across the fleet and defeated correlation; in the Gemini incident, plan-differentiated error codes (429s for one billing plan, 503s for another) disguised a single common origin. Remedy for both: shared identifiers, version manifests, and a common time base as first-class contract terms. I have not yet found a primary source that names distributed tracing and correlation IDs as the remedy; treat that recommendation as practitioner consensus rather than a cited claim.

8. Test the test

One honest line first: no incident in the incident register shows a grader or judge drifting and silently corrupting verdicts. That failure family is evidence-pending. Everything below is derived from the two evaluation-seam contracts (the one gating model artifacts between the model domain and the evaluation plane, and the one gating task-level application behavior at the application seam) and the evidence-package items, which is to say from contract logic rather than from an observed incident. I include it anyway, because the adjacent families (gates that evaluated the wrong thing, verdicts that outlived their evidence) are among the best-attested in the incident register, and the harness is the one artifact nothing else gates.

The harness

The verdict

The escape hatches

9. Start here

If this guide arrived all at once, do not adopt it all at once. Do this, as a worksheet, this week.

Pick three metrics from the section-7 table and instrument the gap between your strongest internal signal and your user reports. For each of the three: write down the internal signal you currently trust most for that outcome. Write down what users would tell you first if the outcome failed anyway. Then measure the distance between the two, in coverage and in time to detection. That gap is where your next incident is already living. Across the incident register, the most expensive failures were announced first by users, quarterly filings, or a tribunal. The seams can be measured before any of those get the chance.

— Maggie Nanyonga