A Timeout Doesn't Tell You Whether the Write Happened

You call an external API to create a ticket. Thirty seconds pass. Your HTTP client gives up and raises a timeout.

What do you know?

You know your process stopped waiting. That's the entire list.

There are at least three worlds consistent with what you just observed. In the first, the request never reached the provider — a DNS failure, a dropped connection, a load balancer that shed it. Nothing happened. In the second, the request arrived, the provider created the ticket, fired its notifications, and the response was lost on the way back to you. In the third, the request arrived and is still being processed right now, and will commit in four seconds, well after you've decided it failed.

Your code has to pick one, and it has no information with which to pick.

  client                             provider
    │                                │
    │  POST /tickets ──────────────> ?
    │                                ?
    │                                ?   ⏱ 30 seconds
    │                                ?
    │  timeout                       ?
  now what?    ┌─────────────────────┴─────────────────────┐
               │ never arrived          → retry is safe    │
               │ arrived, committed     → retry duplicates │
               │ arrived, still running → retry races      │
               └───────────────────────────────────────────┘

I've written about recovering agent workflows after a crash, and that article deliberately walked past this problem — it says some external writes can't be made idempotent and moves on. This is the article that pays that off. Most of what follows comes from building connectors on an internal enterprise assistant, as one implementer in a cross-functional team, where agents could read from and write to several business systems.

The retry is where the damage happens

A timeout on its own is harmless. It's the next line of code that hurts you.

Retrying is a bet that you're in world one. If you're in world two, you have just created a second ticket — and unlike a botched write to your own database, you can't roll it back. It exists in a system you don't own. It has an ID. It fired a notification to somebody. There may already be a human looking at it and wondering why there are two.

This is what makes external effects categorically different from internal state, and it's why "wrap it in a retry" is a decision rather than a default. Your retry policy for an external write is a statement about which failures you believe are safe to repeat, and if you haven't thought about it, the statement you're making is "all of them."

Reads are the easy case and deserve saying quickly: retry them fairly freely. They still cost money, still consume rate limit, and can still leak data you shouldn't have fetched — but repeating one doesn't change the world. Writes have a truth problem that reads simply don't.

Make the operation declare what it is

The change that helped me most wasn't a clever retry algorithm. It was refusing to let a write operation exist without saying, in code, what happens when it goes ambiguous.

@dataclass(frozen=True)
class WriteOperation:
    name: str

    # What the provider gives you to work with
    idempotency_key: str | None          # None = the provider offers nothing
    response_is: Literal["receipt", "acknowledgement"]

    # How to find out what actually happened
    lookup_prior_effect: Callable[[str], Effect | None] | None

    # What to do about it
    retryable_errors: frozenset[type[Exception]]
    compensate: Callable[[Effect], None] | None
    requires_human_reconciliation: bool

Most of the value is in being forced to fill it in. You cannot register a write tool without answering the questions, and the answers are frequently uncomfortable.

The response_is field is the one people skip. There's a real difference between a provider that returns a durable identifier for the thing it created — a receipt, evidence the effect exists — and a provider that returns 202 Accepted and a promise, which tells you it received your intention and nothing about the outcome. Treating an acknowledgement as a receipt is how you record success for something that later failed silently.

And lookup_prior_effect is the field that turns an unanswerable question into an answerable one, which brings me to the actual strategies.

Five options, roughly in order of preference

Use the provider's idempotency key. If the API supports one — Stripe-style, where you generate a key and repeated requests with that key return the original result instead of acting again — use it, and derive the key from your own business identity rather than randomly, so a retry after a process restart still produces the same key. This is the best available answer and it's less common than you'd hope. Many enterprise APIs offer nothing.

Query before you retry. If you can search the provider for an effect matching your business key, do that first. Create-ticket becomes: look for a ticket carrying this reference; if it exists, adopt it and record its ID; if not, create it. This converts "I don't know what happened" into "let me go and find out," and it works even when the provider has no idempotency support at all. The cost is an extra round trip and a search that has to be reliable enough to trust.

Deduplicate on your side. Give the logical operation a business key you control, and let your own database enforce that it happens once. This is the one I've had most success with, and it's the next section.

Compensate. Undo the effect if you discover you duplicated it. Sometimes possible — delete the extra ticket — and frequently not, because notifications already went out and deletion is its own event.

Fail closed and reconcile. Mark the operation as uncertain, stop, and surface it. This feels like giving up and it's often correct. An uncertain payment or permission change should never be resolved by guessing; it should be resolved by a person with the authority to look at both systems.

The worked example: an escalation relay

The clearest instance of the deduplication strategy I built was a human-escalation path, and it's worth walking through because it's small enough to hold in your head.

The setup: a user talking to the assistant asks for a human. The system creates a thread in a support space, links it to the original conversation, and relays messages between them. A chat platform sits in front of all of this — and chat platforms retry event deliveries. Aggressively. If you don't respond quickly enough, or you 500, or the network hiccups, the same event arrives again.

Without protection, a duplicate event creates a second escalation thread for the same conversation, and now two support people are answering the same person in two places.

The first fix is a constraint, but the order matters. You have to claim the logical operation before you create anything in the support system:

CREATE UNIQUE INDEX escalation_conversation_unique
    ON escalations (conversation_id);

INSERT INTO escalations (conversation_id, status, created_at)
VALUES (:conversation_id, 'creating', now())
ON CONFLICT (conversation_id) DO NOTHING
RETURNING id;

The handler that gets an ID back owns creation. The one that gets an empty result reads the existing row and does nothing externally. That turns two concurrent deliveries into one winner before either of them has had a chance to create a thread.

Doing this in the opposite order does not work. If both handlers call the provider, receive different thread IDs, and only then race to insert them, the unique index deduplicates your bookkeeping after the duplicate external effect has already happened.

The winning handler still has a crash window between calling the provider and recording its receipt, so the database claim is necessary but not sufficient. The external call needs the same logical identity:

thread = support.create_thread(
    external_reference=f"escalation:{escalation_id}",
    idempotency_key=f"escalation:{escalation_id}",
)

UPDATE escalations
   SET status = 'active',
       escalation_thread_id = :thread_id
 WHERE id = :escalation_id
   AND status = 'creating';

If the provider supports an idempotency key, a retry converges on the original thread. If it only supports search, query by external_reference before retrying and adopt the thread you find. If it supports neither, leave the row in an uncertain state and reconcile it; do not turn creating back into permission to create another thread.

Then the other half. A human replies in the support thread, and that reply gets forwarded back to the user. Same problem, different shape: the reply event can arrive twice, and two people can reply at nearly the same moment. Only the first should be treated as the answer.

CREATE UNIQUE INDEX delivery_outbox_dedupe_unique
    ON delivery_outbox (dedupe_key);

BEGIN;

WITH winner AS (
    UPDATE escalations
       SET first_reply_id = :message_id,
           first_reply_at = now()
     WHERE conversation_id = :conversation_id
       AND first_reply_id IS NULL
    RETURNING id
)
INSERT INTO delivery_outbox (
    dedupe_key,
    escalation_id,
    message_id,
    status
)
SELECT
    'escalation-first-reply:' || id,
    id,
    :message_id,
    'pending'
FROM winner
ON CONFLICT (dedupe_key) DO NOTHING;

COMMIT;

The conditional update still chooses one reply, but the same transaction also records the obligation to deliver it. A crash before commit leaves neither change behind, so redelivery can try again. A crash after commit leaves a pending outbox item, so a worker can resume delivery instead of deciding the reply was already sent.

The outbox worker uses dedupe_key as the provider's idempotency key and records the returned receipt before marking the item sent. If the provider cannot make that final send idempotent or discoverable, the timeout remains an uncertain delivery that has to be reconciled. The outbox closes the local crash gap; it does not repeal the three-worlds problem at the network boundary.

None of this is clever. That's the point. A unique index, a conditional insert, a conditional update and a durable outbox replace an entire class of "did we already do this?" reasoning that would otherwise live in application code and be wrong under concurrency.

Your dedupe key is not the provider's key

One trap worth naming, because it's tempting and it bites late.

When you're deduplicating on an external event, the obvious key is the identifier the provider gave you. Use it as your primary key and you've bound your schema to somebody else's identifier scheme — one you don't control, can't version, and can't guarantee the uniqueness properties of. Provider IDs get reused across workspaces, change format between API versions, and are sometimes scoped in ways the documentation doesn't mention.

Map external identifiers to internal ones instead. The provider's thread ID becomes an attribute of your conversation record, not the identity of it. Your own ID is what everything else references. It costs one lookup and it means a provider changing its identifier format is a migration rather than an incident.

Acknowledge fast, execute slowly

Here's a failure loop that's easy to build by accident.

A webhook arrives. Your handler kicks off the real work — which involves a model call, three connector reads and a write, and takes twenty seconds. The provider's webhook timeout is ten. It gives up and retries. Now you have two executions of the same work in flight, and your defences are being tested by traffic you generated yourself.

Separate the acknowledgement from the execution. Validate the signature, write the event down, return 200, and do the work asynchronously. The provider is satisfied in milliseconds and stops retrying, and you've removed yourself as a source of the duplicates you then have to deduplicate.

This matters more with agent workloads than with ordinary CRUD, because agent execution times are long and variable in a way webhook timeouts were never designed around.

The one nobody plans for: concurrent token refresh

A write-adjacent failure I hadn't considered until I hit it.

Two requests are in flight using the same OAuth credential. The token expires. Both notice, both refresh, and one of them wins — and depending on the provider, the winner's refresh may invalidate the token the loser is currently holding. The loser's in-flight write now fails with an authentication error that looks nothing like a token problem and everything like a permissions bug.

Worse, it fails after it may already have been received. So now you're in the three-worlds situation, with a misleading error class on top.

The fix is boring — serialise refreshes per credential, treat the refresh as a write in its own right — but the reason I mention it is that "our connector is intermittently unauthorised under load" is an extremely hard thing to diagnose if you haven't already thought about token lifecycle as a concurrency problem.

The questions I ask before shipping a write

Compressed from a longer list I ended up carrying between projects:

1. Is this operation read-only, reversible, idempotent, compensatable — or genuinely ambiguous?
2. Does the provider support an idempotency key, and can I derive it from business identity rather than randomly?
3. Is the response a receipt, or only an acknowledgement?
4. Can I query the provider afterwards to discover what actually committed?
5. Which errors are safely retryable, and which are lying to me about being transient?
6. What's the business key, and does my own database enforce it?
7. How do duplicate and out-of-order events behave?
8. What happens when a token refresh succeeds for one concurrent request and invalidates another?
9. When does this stop being an engineering problem and become a human reconciliation problem — and who gets told?

If a write operation can't answer the first three, it isn't ready to be exposed to an agent. Agents make more calls than humans do, they make them faster, and they retry without getting bored.

The rule

A timeout is not a failure. It's the absence of information, and the only real mistake is treating absence as evidence.

You can't remove the ambiguity — the network will always be able to lose your response after the work is done. What you can do is arrange things so the ambiguity stops mattering: a key the provider honours, a query that tells you what happened, a constraint in your own database that makes the second attempt converge on the first, or an honest stop with a human in the loop.

What you can't do is retry and hope. That isn't resilience. It's just a duplicate you haven't found yet.