The AI Spend Blast-Radius Checklist for Production Teams

The dangerous AI spend incident is rarely caused by someone choosing an expensive model on purpose. More often, a normal engineering failure---a retry loop, a replayed queue, a leaked key, or an agent that never reaches a terminal state---keeps making legitimate calls long after the team has stopped looking.

The question to ask is not "Do we have a budget?" It is: what is the maximum cost of a failure before a person or an automated control stops it? That maximum is your blast radius.

Use this checklist before the next background job, autonomous workflow, or multi-provider rollout.

An unbounded retry loop

Failure pattern: A request fails for a transient reason, and retry code repeats it without a strict attempt limit, backoff, or circuit breaker. With an LLM, each retry can replay a large context and tool chain.

Controls to put in place:

• Set a maximum number of attempts per job and per user action.
• Use exponential backoff with jitter; do not retry every error class.
• Record an idempotency key before a job makes a billable call, so a completed side effect is not repeated.
• Cap the cost or token budget for a single job, not merely the account's monthly spend.
• Alert on retry count as well as spend. A rising retry count is often the earliest useful signal.

Question for the on-call owner: If this one job retries for six hours, how much can it spend---and what disables it?

A queue replay or cron job that runs twice

Failure pattern: A worker restarts, an acknowledgement is lost, a deployment schedules duplicate work, or a cron expression catches up after downtime. Every item is processed again.

Controls to put in place:

• Make every expensive task idempotent, with a durable completion record.
• Set a concurrency and dollar limit for each queue or scheduler.
• Put a human-review gate on unusual catch-up volume.
• Track completed tasks, cost per task, and provider requests together. A healthy request success rate does not prove the work ran only once.
• Give each background job its own API key or project when your provider permits it; that makes a ceiling meaningful.

Question for the on-call owner: Can we distinguish normal batch volume from a replay in one dashboard or one alert?

An agent that does not terminate

Failure pattern: An agent keeps re-planning, calling a tool that returns an ambiguous error, or reloading growing conversation context. It may appear to be "working" while it performs no useful progress.

Controls to put in place:

• Set maximum steps, wall-clock time, tool calls, and token/cost budget for every run.
• Require a meaningful state change before the agent can repeat a tool call.
• Keep a compact task state instead of appending unlimited raw conversation history.
• Emit a terminal reason: completed, blocked, budget exhausted, step limit, or timeout. "Still running" is not a terminal state.
• Alert on repeated tool signatures and on cost per successful outcome, not only total spend.

Question for the product owner: What condition proves the agent made progress on its last five steps?

A provider fallback that silently becomes expensive

Failure pattern: A preferred model or provider is unavailable, so routing falls back to a higher-priced model, a different region, or a path that bypasses caching. The requests succeed, but the unit economics change.

Controls to put in place:

• Declare an allowed fallback list by feature, including a maximum price per request or per task.
• Log the chosen model, provider, and fallback reason with every request.
• Alert if fallback volume or average cost per successful request crosses a threshold.
• Review routing changes as production changes, with an owner and rollback plan.

Question for the platform owner: What is the most expensive route a single customer action can take today?

A leaked or mis-scoped key

Failure pattern: A production key appears in a client bundle, log, repository, or an overly broad integration. A third party can make valid calls until someone notices.

Controls to put in place:

• Give applications independent, least-privilege keys; do not share an owner credential with production code.
• Set separate key ceilings and alerts for every environment.
• Rotate keys with a documented runbook and test the revocation path before an incident.
• Keep a small, explicit list of people and systems that can create or delete keys.
• Treat a sharp after-hours spend increase as a security event until disproven.

Question for the security owner: Which key can spend the most money before a human has to intervene? Read more about how ModelBrake protects credentials.

Context bloat and accidental work amplification

Failure pattern: Each agent step carries a larger transcript, duplicated retrieval context, or repeated tool payload. The task still works, but its cost rises nonlinearly with session length.

Controls to put in place:

• Measure input and output tokens by workflow stage, not only by model.
• Cap retrieved documents and context size; summarize state deliberately.
• Deduplicate repeated tool calls and cache deterministic results.
• Define a cost-per-successful-task budget and alert when it shifts materially.
• Test worst-case inputs in staging, including long documents, empty search results, and repeated failures.

Question for the engineering lead: Which part of this workflow gets more expensive every time it loops?

The dashboard is the only backstop

Failure pattern: The team has a provider dashboard and perhaps a monthly budget, but no one sees it during an incident. Billing data is useful, yet availability and granularity vary by provider.

For example, OpenAI exposes both usage and cost information programmatically, but notes that usage and costs may not reconcile perfectly; it recommends the Costs endpoint or dashboard costs for financial reconciliation. OpenAI documentation AWS Cost Explorer is valuable for analysis but AWS says current-month data is typically prepared on an approximately daily cadence, so it is not minute-by-minute incident telemetry. AWS documentation Vercel AI Gateway can enforce budgets before requests and send threshold emails, which is a strong native control for traffic routed through the Gateway. Vercel documentation

Controls to put in place:

• Define who receives an alert, who owns the first response, and how they can stop further spending.
• Use the strongest native cap available, then add an independent alerting path for the providers and keys it does not cover.
• Run a monthly tabletop exercise: simulate a key that has exceeded its ceiling and confirm the person on call can revoke or disable it.
• Be honest about detection latency. A control that polls billing data is valuable, but it is not the same as per-request enforcement.

Question for the incident commander: At 2 a.m., will a person receive a signal that is actionable before the next billing cycle?

A five-minute scorecard

Score each item as green, yellow, or red:

Check	Green	Yellow	Red
Per-job cost limit	enforced	documented only	absent
Retry controls	capped and observable	capped but not observed	unbounded
Agent termination	steps, time, and budget capped	only one cap	absent
Key isolation	per app/environment	partly shared	shared owner key
Fallback cost guardrail	enforced	monitored	unknown
Independent spend signal	tested alert and owner	dashboard only	none
Stop mechanism	tested and documented	documented only	unclear

Two or more red rows is not a reason to panic. It is a reason to choose the first failure mode to contain this week.

Where ModelBrake fits

ModelBrake is designed as an independent spend backstop for OpenAI, Anthropic, AWS Bedrock, and Vercel. It uses provider billing data with a read-only monitoring credential; it does not route application traffic or see prompts. It can alert on abnormal spend and per-key ceilings, and an optional separately scoped credential can be used for a deliberate revocation workflow.

That makes ModelBrake a complement---not a replacement---for safe retries, idempotency, native provider caps, and request-level observability. If a team needs per-request traces, routing, or enforcement before a request leaves its service, a gateway or proxy is the appropriate layer. If it needs a low-touch, independent circuit breaker for spend, ModelBrake is built for that job.

Want a second set of eyes on your blast radius? Get a 15-minute spend-risk audit.