Retrying an AI Agent Is Not the Same as Retrying an API Call

The worker that crashes is not the one that will hurt you. The one that will hurt you is the worker your system has already declared dead, which is still very much alive and about to write its results.

Here is the shape of it. A worker is halfway through generating a long research report. It stops heartbeating — GC pause, network blip, a provider call that hangs for four minutes, it doesn't matter why. Its lease expires. Another worker sees an abandoned job, picks it up, and resumes from the last checkpoint. And then the first worker wakes up, finishes what it was doing, and tries to save.

Now suppose you sort that out, because ownership is a solved problem in distributed systems and you can just look up how to do it. There is a second broken state machine waiting inside the agent itself. Its message history might end with an assistant message asking for a tool, with no tool result recorded. Hand that history back to the provider and the next call is invalid. Replay it instead and you may redo expensive work — or, if that tool writes to something, redo an action out in the world whose outcome you were never sure about in the first place.

“Just retry it” is quietly asking three different questions:

• Who owns this job now?
• How much of the work can I prove is finished?
• Which parts of this conversation are still safe to reuse?

An ordinary API retry replays an input. An agent retry has to recover a workflow.

I worked all of this out on Research-AI, my open-source deep research system, over several months of getting it progressively less wrong. The first version of the retry policy was, essentially, a loop around the entire graph. What it became is two separate recovery layers: one for durable workflow state and ownership, and one for repairing the agent's tool-calling conversation.

The thing I would put on a wall, if I put things on walls:

A retry policy isn't a number. It's a claim about which state you still trust after a failure.

In the beginning, the lifetime was the HTTP request

Early Research-AI ran deep research inside the chat request. The API created the agent, awaited the research graph, and returned the finished document in the same response.

It was simple and it worked in development, which is exactly how these things survive longer than they should. But it tied a workflow that could run for many minutes to the lifetime of one HTTP request in one API process. If either went away, there was nothing left for anyone to continue. No durable unit of work existed.

In February 2026 I moved research behind a background job. The chat endpoint started writing a job document to Firestore and returning a task ID instead of blocking; a worker polled for queued work, ran the graph, and stored the result; the frontend polled for status.

That was a real improvement. It also moved the failure rather than removing it.

The first worker looked roughly like this:

job = find_queued_job()
mark_running(job, worker_id)

try:
    result = run_entire_research_graph(job)
    mark_completed(job, result)
except Exception:
    requeue_with_backoff(job)

Four problems are hiding in those eight lines, and it took me a while to see all of them.

The claim is a query followed by an unconditional update, so two workers can both read the same queued job before either write lands. A process that dies after setting running leaves the job stuck there forever, because everyone else is only looking for queued. None of the completion, failure or requeue writes check that the writer still owns the job. And retrying means re-running every phase that already succeeded, including the expensive ones.

A queue gave me durability for the request. It gave me nothing for recovery.

The job document became a recovery protocol

The change in thinking that mattered most was to stop treating the job record as status bookkeeping and start treating it as the authoritative statement of what is allowed to happen next.

The fields that carry that weight now:

status
workerId
lastHeartbeatAt
leaseExpiresAt
attempts
nextRunAt
error
graphState
resumeFromNode

Each one is answering a specific recovery question. status and nextRunAt decide whether the work is eligible to run at all. workerId records who owns it, and the lease says when somebody else becomes entitled to take it away. attempts and error carry the retry history. graphState is what is durably finished. resumeFromNode names the next boundary worth evaluating.

The state machine on top of that is deliberately small:

queued   -> running    on a conditional claim
running  -> running    on heartbeat or checkpoint
running  -> running    when an expired lease is conditionally reclaimed
running  -> queued     on a retryable error, if budget remains
running  -> failed     on a terminal error or exhausted budget
running  -> completed  on an owner-checked completion

Six transitions. Recovery is difficult enough without a dozen ambiguous intermediate states to reason about, and every bit of real complexity belongs in the transition rules rather than in the diagram.

A lease does not stop the old worker

A worker claims either a due queued job or a running job whose lease has expired, using an optimistic conditional write. In Firestore that means conditioning the update on the document version you read. Several workers can read the same candidate; only one whose snapshot is still current will succeed in changing it.

The winner writes its worker ID, flips the job to running, and stamps an expiry on the claim. While it works it heartbeats — every 60 seconds by default, against a 180 second lease — and each heartbeat pushes the expiry out.

That covers the ordinary case. The interesting case is the delayed worker:

1. Worker A claims the job and starts generating.
2. A stops heartbeating for long enough that its lease expires.
3. Worker B conditionally reclaims the job and resumes from durable state.
4. A carries on running, because a lease expiring somewhere else cannot kill a process.
5. A eventually tries to publish progress, or completion.

Step five is the one that decides whether your data survives.

So every authoritative write a worker makes — progress, checkpoint, requeue, failure, heartbeat, completion — goes through one gate:

def owned_write(job_id, worker_id, changes):
    snapshot = jobs.document(job_id).get()
    job = snapshot.to_dict()

    # Am I still the owner, and is this job still live?
    if job["status"] != "running" or job["workerId"] != worker_id:
        raise OwnershipLost(job_id)

    # Nobody else has touched the document since I read it.
    jobs.document(job_id).update(
        changes,
        option=client.write_option(last_update_time=snapshot.update_time),
    )

Two conditions, and both are load-bearing. The first says I am the recorded owner. The second says nothing changed under me between reading and writing — without it, Worker B could reclaim the job in the gap between Worker A's check and Worker A's update, and A's stale write would land anyway.

Every terminal write goes through that function. There is no path that sets completed without proving ownership first, which is the only reason a delayed worker can't overwrite a newer one's result.

The worker runs its research task alongside its heartbeat task, so the moment the heartbeat notices it has lost ownership, it cancels the research task. Ownership loss discovered through a progress or checkpoint write ends up down the same path.

I want to be honest about what this actually guarantees, because it is easy to overclaim here. The invariant is not “only one worker will ever execute this job.” Network partitions and delayed failure detection make that promise impossible to keep, and pretending otherwise would just move the bug somewhere I couldn't see it. The invariant is narrower: only the current recorded owner may advance the durable job or commit its terminal result.

Leases detect probable abandonment. Owner-checked writes are what stop overlapping computation from becoming conflicting durable state.

I stopped checkpointing where the program was

The research graph has four broad phases:

document outline
    -> expert perspectives
    -> expert content
    -> final document sections

You cannot meaningfully resume at an arbitrary Python instruction inside one of those. The call stack is gone, the sockets are gone, the browser is gone, the in-memory agent threads are gone. What can survive is semantic state — a validated outline, the chosen perspectives, the sections that are actually finished, the rolling summaries.

So after each top-level phase, the graph serialises that state and records the next phase. When a job is reclaimed, the new worker rebuilds typed state from the stored payload. The requested resume node is treated as advisory: if the stored state is missing something that phase needs, recovery walks back to the earliest phase it can safely execute. I would rather redo a phase than build on a checkpoint that only looks complete.

Top-level checkpoints turned out to be far too coarse, though. Expert content is the longest phase by a distance — several experts, each writing several sections — and losing all of it because the process died near the end makes “resumable” technically true and practically worthless.

So I added section-level progress. Experts run concurrently; the sections within one expert run in order, so a later section can use a rolling summary of what came before it. After each section, the system adds its status, content and updated summary to shared checkpoint state and schedules a durable flush. On recovery, a fully completed expert is reused as-is, a partially completed expert restarts at its first unrecorded section, and the final document picks up at its first unrecorded section.

Getting parallel experts to checkpoint safely took two attempts. My first version put a lock around each shared-state update and its Firestore write, which was correct and also meant every section completion had to wait on persistence before the next one could start. I replaced it with a flusher: it merges the latest progress, coalesces revisions, and does the durable writes through a single background path while the experts keep working.

Which gave me a definition I actually like: a checkpoint is the latest coherent unit of work the next worker is allowed to trust. It is emphatically not a promise that nothing after it happened. Anything completed after the last successful checkpoint may well run again.

One caveat I'll flag rather than bury: checkpoint transport is currently best-effort. Ordinary write errors and callback timeouts don't stop the graph. That favours availability, and the price is that a crash can roll recovery back further than the most recently completed section.

Two kinds of retry, for two kinds of state

Once job recovery could resume properly, I went back to failures happening inside a section, and found they needed a completely different policy.

A job-level failure asks: can another execution rebuild this workflow from durable state? That gets a bounded retry budget, exponential backoff, and the last valid resume boundary — and errors classified as terminal skip the loop entirely.

A section-level failure asks something much narrower: can this agent continue the tool-calling conversation it was in the middle of? That state is more detailed, more fragile and much shorter-lived than the durable job state.

Research-AI gives a section one initial attempt, up to two warm retries, up to two cold retries, and then records the section as skipped.

Those five attempts are not five identical calls. Warm and cold make genuinely different trust decisions, and the difference between them is most of the point of this article.

The second state machine lives inside the transcript

Tool-calling history is not ordinary chat text. An assistant tool request and its matching tool result are a protocol exchange, and providers will reject a history where one is missing its partner.

Consider:

Human: Write the reliability section.

AI: call search(query="worker leases")
Tool: search results

AI: call scrape(url="https://example.com/article")
[timeout before the tool result is recorded]

The first exchange is complete. The second isn't.

Real failures are messier than that. One assistant message can request several tools and only some results come back. A final assistant response can be recorded halfway. The context can blow past the model's limit while a call is still unresolved.

Every naive option is wrong in its own way. Append the section prompt again and you have a duplicate instruction sitting after an invalid suffix. Keep the whole transcript and the provider rejects it, because there is a tool call with no result. Replay every tool call and you may repeat expensive work, or an external effect whose outcome you never established.

And that last one is the part people skip: a timeout only proves the caller stopped waiting. It proves nothing about what the tool did.

Which is why the retry mechanism has to look at message structure. Catching the exception is not enough information to decide anything.

Warm retry: keep the valid prefix, repair the rest

A warm retry is a statement that the conversation is still worth keeping, once its failed turn has been cleaned up.

The sanitation works like this. Find the latest exact prompt for the current section, and keep everything up to and including it. Then walk forward through what follows, keeping only complete assistant-tool-call and matching-tool-result groups, and stop at the first message that is incomplete or unexpected. Verify that nothing is left orphaned in either direction — no tool response without its call, no call without its response. Replace the thread history with that prefix, and continue the same thread without appending the section prompt a second time.

Roughly:

safe = history.through(latest_section_prompt)

for exchange in history.after(latest_section_prompt):
    if exchange.is_complete_tool_cycle():
        safe.extend(exchange)
    else:
        break

assert no_unresolved_tool_calls(safe)
retry_same_thread(safe, append_prompt=False)

Note what this doesn't do: it never replays the completed tool calls. It keeps their recorded results and lets the agent carry on from there. Those searches and scrapes already happened and already cost money; running them again to get back to the same place would be absurd.

Context-window failures need one extra step. Strip the unsafe suffix first, then try to compact what remains — compaction is only attempted once sanitation has produced a valid tool-call state, because compacting a broken transcript just gives you a smaller broken transcript. If compaction can't complete, the cleaned history gets inspected again before the warm retry proceeds.

If the thread can't be inspected, rewritten, or shown to be structurally sound, the warm path gives up and hands over to a cold retry. It does not guess.

Cold retry is an isolation boundary

A cold retry starts a new thread generation for that section. The failed thread's detailed history is abandoned, and the section prompt plus the rolling context from completed work are rebuilt in a clean namespace.

The distinction is worth stating plainly. Warm retry says I still trust this conversation. Cold retry says I trust the task and its inputs, but not the conversation that attempted them.

There's a boundary here I should be explicit about. The section agents use an in-memory checkpointer, so warm and cold thread histories only exist inside the running process. If the process dies, durable job recovery does not resurrect the exact tool transcript — it rebuilds the work from completed sections and stored summaries. That is deliberate semantic recovery, not instruction-level continuation, and I think it's the right trade for this workload.

When both budgets are exhausted, the section is recorded as skipped and the rest of the report carries on. At aggregation time, other experts' contributions for that section can still be used. If no expert produced anything for a section, the system writes an explicit fallback rather than quietly pretending the research exists.

That last decision is a product call, not a universal reliability rule, and I want to be careful not to present it as one. A partially complete research report is usually more useful than no report. A payment, a deletion or a permission change should fail closed when its prior outcome is uncertain — the asymmetry is the whole point.

Bounded partial success beats an infinite retry loop. But it is only honest if the partial state is visible to the user.

What this actually guarantees, and what it doesn't

Compared to the request-bound version, Research-AI now has properties I can state with a straight face. Work outlives the HTTP request that created it. Expired jobs get reclaimed instead of sitting in running forever. Workers can't blindly overwrite each other's job state. Recovery reuses completed phases and sections. Same-process retries only reuse structurally valid conversation history. And exhausted retries produce an explicit terminal or partial state rather than silence.

It does not give exactly-once execution. It does not prove that no model call, search, scrape or computation is repeated after a checkpoint. And Firestore polling is a pragmatic queue for the traffic this project sees, not a managed queue with real backpressure, dead-letter handling or delivery guarantees.

There are five things I would fix next, in roughly this order.

I'd replace the process-scoped worker ID with a unique claim token, or a monotonically increasing lease epoch. Today, if one long-lived worker somehow reclaimed its own expired job while the original coroutine was still alive, both tasks would share an owner identity and the ownership check would wave both through. A per-claim token fences the older execution properly.

I'd make terminal publication idempotent. The finished report is appended to chat history before the owner-checked completion transition, and a narrow crash or ownership race in that gap can duplicate the final message. Publishing with the job ID as an idempotency key, or through an outbox committed alongside the terminal transition, closes it.

I'd turn the configured batch size into an actual concurrency limit. The worker can claim another batch while earlier tasks are still running, which means the batch size is currently a suggestion. A semaphore, and claim capacity derived from free slots, would make backpressure explicit instead of emergent.

I'd track stale reclaims separately from ordinary caught exceptions. A hard process crash doesn't currently consume the normal retry budget, so a job that reliably kills its workers needs its own poison-job policy rather than being retried forever by well-behaved machines.

And I'd build a fault-injection suite around the transitions: crash right after a tool returns, expire a lease mid-generation, fail a checkpoint write, deliver only part of a multi-tool result set, and let an old worker attempt completion after a reclaim. Recovery code is those interleavings. Happy-path tests cannot tell you anything about it.

None of these gaps make the design useless. They describe where its guarantees actually stop, which is more useful than a claim of exactly-once that falls over the first time two workers overlap. The honest description is at-least-once computation, with owner-checked durable transitions and semantic resume.

The rules I'd carry into the next system

The implementation changed several times. What survived is short:

1. A retry has to begin at an explicit recovery boundary.
2. Every authoritative workflow write must prove current ownership.
3. A checkpoint is a coherent, validated unit of completed work — not “the worker wrote something recently”.
4. Never continue a transcript that has unresolved tool calls.
5. Keep same-thread retry and clean-thread reconstruction as separate mechanisms.
6. Assume anything after the last durable checkpoint will run again.
7. Give exhausted retries an explicit terminal or partial state.
8. Reconcile external effects. Cleaning up a conversation is not idempotency.

Retrying an agent is not a loop around an LLM call. It's a recovery protocol spanning durable workflow state, temporary worker ownership, and the state machine hiding inside the tool-calling transcript.

I used to start with “how many times should this retry?”

Now I start with: what can I prove completed, who is still allowed to continue, which state am I willing to trust on the next attempt, and what might already have happened outside my process?

Until those have precise answers, another retry isn't resilience. It's just another chance to corrupt the work.