How to Run an AI Spend Incident Review

An AI spend incident is a production incident. The right review is blameless, specific, and useful: it identifies the path that turned a normal defect into billable work, confirms what contained it, and assigns improvements with owners and dates.

The one-page incident record

Summary

• Incident title:
• Date and time window (UTC):
• Affected provider, organization/project, and key:
• Estimated incremental cost:
• Customer or service impact:
• Incident commander and reviewers:

What happened

Describe the system behavior in plain language. Example: "A deployment retried a failed enrichment job without a bound. The worker replayed 18,000 tasks with a long context payload until the queue was paused." Do not start with a person's name or an assumption about intent.

Reconstruct the timeline

Time	Event	Evidence	Decision / owner
	first abnormal request or job	logs / metrics
	first cost or anomaly signal	provider / monitor
	alert delivered	email / Slack / pager
	human acknowledgement	incident channel
	containment	queue paused / key disabled
	service restored	deploy / config change

The most valuable numbers are detection delay, time-to-containment, requests or tasks after the first signal, and incremental cost after the first signal.

Identify the failure chain

Answer each question with evidence:

1. What initiated the work?
2. Why did the work continue instead of terminating?
3. Which retry, queue, agent, fallback, or credential control failed or was absent?
4. Why did existing monitoring not produce an actionable signal sooner?
5. Who had authority to stop the work, and was that path clear?
6. What stopped the incident in practice?

Do not collapse "root cause" into one line. A billing event usually needs both a triggering defect and missing containment.

Corrective actions

Action	Failure mode addressed	Owner	Due date	Success test
Set maximum retry count	unbounded retries			staged test terminates at limit
Add per-job dollar cap	large single run			simulated run stops or alerts
Separate production key	broad blast radius			key maps to one workload
Add anomaly recipient	late detection			alert acknowledged in test
Rehearse key revocation	slow containment			staging key is disabled and restored

Close an action only when its success test has run. A ticket that says "add alerts" is not complete until the alert reaches an owner who can act.

Prevention review cadence

Review the top AI-spend workload each month, whether or not an incident occurred. Revisit ceilings after model, routing, pricing, queue, or deployment changes. The incident review should make the next surprise less likely, not merely explain the last one.

ModelBrake can provide an independent signal when provider spend breaks from a baseline or crosses a per-key ceiling. Its value is greatest when the team has already decided what happens after the signal.

Use this template after your next cost anomaly---or as a tabletop exercise before one occurs.