Postmortem Template for AI Agent Outages

A postmortem author stares at the "Root Cause" field after an agent outage and has nowhere to put the actual answer. The Five Whys method, the backbone of most postmortem practice, asks the team to walk down a chain of "why" questions until it lands on something that can be diffed: a line of code, a config file, a process step that someone skipped. That structure turns an incident into a list of changes to make, and for decades it has worked well enough because most incidents really do trace back to something a team can point at and alter.

That structure breaks the moment one of the actors involved is an inferential system whose output is sampled from a distribution. Faced with a template that has no field for that answer, the postmortem author does the only thing the form allows: they map the failure onto the nearest available category. The same class of incident recurs, and it gets written up the same way again, because nothing in the template forced anyone to name what actually happened inside the model's reasoning.

What makes agent failures categorically different from the failures templates were designed for

An AI agent incident is not a single malfunction. The operational record and the behavioral record are simply not the same record, and a template that only knows how to ask about the former has nothing to say about the latter.

The same visible output, the same final answer shown to a user, can trace back to wildly different causes depending on which system prompt was active, which model and parameters generated it, which tools were defined and available, which documents got retrieved, what the conversation history contained, which routing path the request took, which policy version was in force, and what side effects fired along the way. Speed compounds this: an agent can spawn sub-agents, call a dozen APIs in sequence, and complete an irreversible action, issuing a refund, deleting a record, sending a message to a customer, in less time than it takes a human on call to open a ticket. Many of these effects also cannot be undone by rollback. Recovery on the infrastructure side tells a team almost nothing about whether the underlying behavior has been fixed.

These failures sort into categories that a deterministic template has no language for: an inference error, where the agent read the situation correctly but chose the wrong plan and the runbook itself was fine; a permission scope failure, where the agent held credentials broader than the task needed and that excess scope widened the damage once something went wrong; a missing or bypassed approval gate, where the control existed on paper but was never enforced at the moment it mattered; a cascading multi-agent failure, where specification mismatches and weak verification between agents compound across a pipeline; and memory poisoning, where stale or incorrectly summarized context propagates an error through every step that relies on it. What a usable template needs, then, is not a new row bolted onto the old form but two separate evidence planes running in parallel: one that tracks the operational state of the system, and one that tracks what the agent actually believed and did.

What the two evidence planes must preserve

The operational plane is the plane most incident response tooling already understands well: it covers availability, latency, and exceptions; deployment and configuration events; logs and infrastructure traces; the alert and recovery timeline; and the point at which service was restored. None of this is new, and none of it should be discarded, because an agent still runs on infrastructure that can fail in ordinary ways.

Most templates lack a behavioral plane. Even detection itself can misfire in ways that look correct, and a successful recovery on the operational side proves nothing about whether the behavioral problem was actually addressed.

Reconstructing which agent acted, under whose authority, with what context, through which tools, and what changed as a result depends on a single trace ID per run that links every model call, tool call, policy decision, and approval together, along with before-and-after values for any record the agent touched. The behavioral plane is where the first causal divergence actually lives, and everything that follows in this template exists to make that plane recordable rather than reconstructed after the fact from memory and guesswork.

Identity and ownership: the fields that make the postmortem traceable

Before any evidence gets recorded, a postmortem needs a permanent identity: an incident ID, a short title naming both the behavior and the impact, a status, an incident owner, a technical reviewer, the severity rule that was applied, and a durable record URL with one person accountable for it.

Alongside that administrative spine, the team needs to reconstruct the active artifact IDs for the incident: trace_id, session_id, bundle_id, change_id, rollout_wave, policy_bundle, approval_mode, and tool_principal. The EU AI Act's Article 73 requires structured incident reporting that includes investigation, risk assessment, and corrective action, and a postmortem that cannot be traced to a specific agent version and configuration simply cannot meet that bar. Identity fields make the behavioral evidence attributable to a specific run and make any later corrective action verifiable against that same run.

User and business impact: what went wrong for the people affected

Standard severity rules built around uptime percentages were not designed to catch what agent outages actually do to the people on the receiving end, and a postmortem that only records service degradation will undercount the real harm and misclassify the incident's severity.

Irreversibility is the field that does the most work here. An AI agent incident crosses the threshold that demands reporting when it causes user harm, an incorrect action that cannot be undone, a material violation of policy, sustained failure to complete a task, or loss of required human control over the system, and a large share of these effects cannot be fixed by rolling back a deployment. Where a number cannot actually be measured, the postmortem should say so directly. Writing UNOBSERVED where a measurement is missing, rather than writing a zero or a best guess formatted as if it were observed, is a more honest account of what the team actually knows and a more useful one for whoever reads the record later.

Expected and observed behavior: the section that replaces "root cause" for inferential systems

This is the section that replaces root cause analysis for a system that reasons rather than executes fixed logic, and it is where the template earns its purpose. The baseline fields are the expected outcome, the observed outcome, the specific policy or product contract that was violated, and a representative trace tied to the rule it broke. What makes the section work are four additional fields that standard templates have no place for.

The first is a direct statement of what the agent inferred: its actual reading of the situation, pulled verbatim from the trace rather than paraphrased by whoever is writing the postmortem. The second is what the correct inference would have been: the factual reading the agent should have reached, which defines the actual gap the corrective action needs to close. The third is the specific signal the agent missed, the piece of context that, had it been visible, would have changed the inference. "Improve the prompt" is the new version of "fix the runbook": it sounds like an action item while quietly shifting the burden onto the model's reasoning rather than onto the information environment the team actually owns.

Together, these four fields move the center of gravity of the investigation off the model call itself and onto the context pipeline, the tool surface, and the escalation policy surrounding it, which are the parts of the system a team can actually redesign. Naming what the agent believed also does something further: it determines which root-cause category the incident belongs to, which is the question the next section exists to answer.

Execution lineage: reconstructing the system that ran

Behavioral evidence only means something when it is pinned to the exact system that produced it, because the same prompt and the same model can behave differently depending on the tools available, the documents retrieved, the permissions granted, and the deployment configuration active at the time. The lineage section should record trace and run IDs, the path the agent took, the prompt version by immutable ID or hash, the model and its parameters, the tool definitions and their versions, the permissions that were actually granted, the retrieved documents and where they came from, the conversation history, a marker for the deployment in effect, and any feature flags that were on.

Permission scope deserves to be treated as its own forensic artifact. A paragraph of prose describing what should have happened cannot substitute for those fields being present or absent in the actual record.

For systems where multiple agents hand work off to one another, the postmortem needs to record which agent handed off to which, tied together by the session and trace IDs that link the chain. Every field in this section is also a gap when it is missing, and naming the gap directly, rather than just leaving the field blank, is what turns the lineage section into something a team can actually use.

First causal step and root-cause category: moving past "the model made a mistake"

With the behavioral evidence and the execution lineage both reconstructed, the postmortem can finally name a first causal step. That means identifying the first supported divergence, the earliest point in the trace where the agent's behavior departed from what was expected, along with the downstream symptoms that followed from it, the causal edge connecting the two, the failure category it belongs to, the type of fix it calls for, an owner, and a stated confidence level. Where a cause cannot be pinned down with confidence, the postmortem should say so explicitly rather than forcing a single root cause onto evidence that does not support one.

Most agent incidents, especially multi-agent and memory-related ones, have more than one amplifying condition, and the postmortem should record those contributing factors. The category assignment should also settle a second question: whether the actual problem lived in policy, approvals, rollout process, memory handling, observability, or inventory management, because that determines which lifecycle artifact needs to change, not merely which ticket gets filed.

The obvious objection to building a categorization scheme at all is that most AI incident databases today are voluntary and catch only a fraction of real events, and the chain-of-thought data that would make these categories easier to assign is rarely shared outside the team that generated it. The objection is worth taking seriously, but incomplete categorization still beats no categorization. Labeling a confidence level honestly and leaving unresolved items marked as such gives the next team something to build on. The Agent Incident Registry codes its incidents against a fixed scheme with a crosswalk to OWASP's Top 10 for Agentic Applications and to MITRE ATLAS, functioning in effect as a shared reference system for agent incidents and giving any team a public baseline to check its own category assignments against.

Containment, mitigation, and recovery: the ordering discipline that protects forensic evidence

In an agent incident, the order of operations matters as much as the operations themselves. Stopping autonomous action has to come before any debate over root cause, because an agent with live permissions keeps changing the risk in real time while the team is still arguing about what happened. Evidence preservation has to come before remediation, because the artifacts that would explain the failure are often the first things a standard incident response process overwrites or deletes.

The containment sequence should move the agent into a disabled, read-only, or human-approval-required state first. Before any remediation touches the system, the evidence that needs to be pulled and preserved includes prompts, tool-call logs, model and version IDs, queue names, IAM events, OAuth grants, and audit trails. Restoring service on the operational plane and confirming a fix on the behavioral plane are two separate checkpoints, and both need to appear in the record. This section of the postmortem should capture the containment actions taken in the first minutes, what was disabled, restricted, or moved into a restricted mode, whether a rollback was required, what harm actually stopped as a result, what behavior persisted even after mitigation, and what failure state, if any, remained open.

Corrective change and validation: connecting a fix to a verified production result

A corrective change only counts as validated once it has been tested against a frozen reproduction of the original failure and then confirmed under the same conditions in production, not simply once the dashboard turns green again. The corrective change itself needs a change ID, an owner, a stated hypothesis, the exact diff applied, the type of fix it represents, a record of its approval, and a defined rollback target in case it does not hold.

Validation needs its own set of fields: a frozen reproduction of the original failure, matched runs between the old baseline and the new candidate, any regression controls applied, a marker for the deployment that carried the fix, evidence gathered from production after the change, and any confounders that might complicate the result. Only once the behavioral fix has been confirmed this way can the postmortem's executive summary actually say what was fixed and what remains unverified, which is the point at which the summary can finally be written honestly.

The minimum checklist for closing a postmortem and feeding it forward

A postmortem for an agent outage is not finished until it can answer seven questions, and anything left unanswered needs to be marked as such. Does the incident feed back into the eval set and into rollout criteria going forward.

Every "no" on that list should be written into the postmortem explicitly. The standard template was built for failures with one cause and one fix, and an agent that infers wrong was never going to fit inside it. A template built on two evidence planes instead of one does not just document the outage more completely. It changes what the team sitting on the other side of the next incident is actually able to learn.