A Production Checklist for Retry Loops That Call LLMs

Retries are not a bug. Unbounded retries are. For an LLM workflow, every retry can re-send a large context, repeat a tool call, or restart an entire agent path. The retry policy needs to be a cost-control policy as well as a reliability policy.

Before retrying, classify the failure

Retry only errors that are plausibly transient: a short network interruption, a rate-limit response with a valid retry-after signal, or a temporary provider error. Do not retry invalid input, missing credentials, authorization failures, unsupported parameters, or a task that already completed its side effect.

Each error class should have an explicit owner and policy. "Catch all and retry" is not a policy.

Set four limits, not one

• Attempt limit: maximum retries for a single call or job.
• Time limit: maximum wall-clock duration for a run.
• Work limit: maximum agent steps or tool calls.
• Cost limit: maximum tokens or dollars for the run.

Use exponential backoff with jitter, and respect provider retry guidance. A quick loop is especially dangerous when multiple workers see the same error at once.

Make work idempotent

Write a durable operation ID before the billable call, and record whether the side effect completed. If the worker restarts, it should resume or safely stop---not create a second charge and a second user-visible action. For tools that make external changes, create an idempotency key at the boundary, not only inside the agent process.

Bound agent retries separately

An agent's "try again" can be more expensive than a transport retry. It may re-plan, re-query retrieval, reload context, and call multiple tools. Track a repeated tool signature, require state change before another attempt, and stop when the agent cannot demonstrate progress.

Observe the right signals

Monitor attempts per job, time in retry, token/cost per successful outcome, queue age, and terminal reason. Alert before total monthly spend becomes the only signal. A small increase in retries can reveal a large future bill.

Test the unhappy path

In staging, simulate a retryable provider error, a non-retryable authentication error, a queue restart, and an agent loop. Confirm that each reaches a terminal state inside its time and cost budget. Confirm that the alert says who owns the job and how to stop it.

ModelBrake can alert when provider spend deviates from normal or exceeds a per-key ceiling. It is a useful independent backstop; the retry policy itself must still live with the code that decides to make the next call.

Use the AI Spend Blast-Radius Checklist to test your current retry policy.