A lost response is not a lost request

The problem

A member's phone lost data for about ten seconds. She was in the middle of writing a community post, tapped Post, and got told it failed. She tapped again. Failed. Again. Again.

When the network came back, four identical posts appeared in the feed.

The obvious suspect is the button — four taps, four posts. But the Post button is disabled the moment a publish starts (postable = state.canPost && !state.busy), and publish() guards on state.posting as a second line of defence. Taps two through four never reached the network at all. Whatever produced four posts, it wasn't her.

Where it actually was

Every network call in the app rides one client with the same discipline: a 15-second timeout, and retry-with-backoff — three attempts, 500 ms exponential base. That policy lives in one function, and the line that decides whether to try again is this:

final retryable = result.networkError || result.statusCode >= 500;
if (result.ok || !retryable || lastAttempt) return result;

Two things about that line, neither of which is visible while reading it.

The first is that it applies to every HTTP method. GET, POST, PATCH, DELETE — the function takes method as a parameter and never consults it when deciding to retry.

The second is what networkError includes. A few lines down:

} on TimeoutException catch (e, st) {
  return const JsonResponse(statusCode: JsonResponse.networkStatus, networkError: true);
}

And the timeout wraps request.close() — the call that sends the request. So the timer can fire after the bytes have already left the device and been processed by the server. All that was lost was the answer.

That's the whole bug, and it's one sentence: a lost response and a lost request are indistinguishable from the client, and the retry treats them as the same thing. One tap became up to three POST /cmty/posts. Some landed. The server had no reason to think they were the same post, because nothing in the request said so.

A 5xx retry carries the identical hazard, for anyone tempted to keep just that half: a 502 from a gateway can arrive after the application server has already committed.

The most useful thing I found while confirming this was in a completely different module. commerce, in the same backend, had solved it already — NewOrder.idempotency_key, an order id derived from it, a conditional create. The answer existed; community had just never been given it.

The fix, and the trap next to it

The naive fix is a key table: store every idempotency key you've seen, check it before writing, expire the rows on a TTL. That's a new table, a new read on the hot path, and a new thing to get wrong.

The commerce pattern needs none of it. Derive the row's primary key from the idempotency key, then let a conditional create do the work:

def _post_id(idempotency_key: str) -> str:
    if idempotency_key:
        return uuid.uuid5(uuid.NAMESPACE_DNS, _IDEMPOTENCY_NS + idempotency_key).hex
    return new_id_str()

A replay computes the same id. And the repository floor already put a guard on every create it builds:

"ConditionExpression": "attribute_not_exists(pk)",

So the second write cancels itself. Nothing new is stored, nothing new is read, and there is no TTL to tune. The missing-pet flow in the same file already depended on exactly this mechanism — the fix was less invention than noticing.

The trap is in what you do with that cancellation. It is tempting to catch it and return the existing row, and that is almost right. A transaction cancels for reasons that have nothing to do with a replay: contention the retry budget gave up on, a guard on some other item in the same batch. Answer those with somebody's earlier post and a genuine failure becomes a confident wrong answer.

Worse: the key is client-minted. Two members who somehow present the same string would derive the same id, and a naive handler would hand one of them the other's post.

So the replay path re-raises unless two things hold — a key was actually supplied, and the existing row belongs to this caller:

if not idempotency_key:
    raise cancelled
existing = await get_post(post_id, viewer_id=viewer_id)
if existing is None or existing.author.id != viewer_id:
    raise cancelled

The other half of the fix is on the client, and it's the part I'd have got wrong if I hadn't re-read her bug report. The instinct is to mint a key per request. That fixes the automatic retries and nothing else — because the member also taps Post again after being told it failed, and that report may itself have been a lost answer.

So the key is minted once per intended post, not once per attempt. It survives all three HTTP retries, it survives every re-tap of the same draft, and it is dropped only when a publish actually succeeds — at which point the next post genuinely is a different post and earns a new one.

How it was verified, not just believed

Four tests on the server, against a real database rather than a mock — a dedupe asserted against a mock proves nothing about the conditional write that performs it:

• the same key twice returns the same post id, and does not raise
• two different keys produce two posts — the dedupe keys on the key, never on the content, because an author may legitimately publish the same words twice
• no key behaves exactly as before: a fresh random id, no dedupe, so an older client is untouched
• a different member presenting the same key is refused, not handed the first member's post

And two on the client, which pin the distinction the whole fix rests on: three attempts at one draft present one key; two posts the author actually meant present two.

That first client test is the one that matters. Without it, "we added an idempotency key" and "we fixed the bug" look identical in a diff and differ completely in production.

One smaller thing surfaced

One smaller thing surfaced in the same investigation, from the same retry policy. Image uploads ride it too, and when all three attempts fail the photo is simply not attached — the staged tile disappears, a three-second toast is the only signal, and Post stays enabled. The post publishes with fewer pictures than the author chose.

That had been reported separately as "I uploaded 2 images, only 1 is visible", and it got its own fix, because a create being safe to repeat and a create being complete are two different guarantees.

Key takeaways

"Transient" describes the network, not the safety of repeating the call. Those are different axes, and a retry policy has to reason about both. Every retryable failure here really was transient; that was never the problem.

A timeout is the dangerous case, not the harmless one. A connection refused before the bytes leave is safe to retry. A timeout means you don't know. Any policy that folds both into one networkError flag has thrown away the only bit that mattered.

Idempotency is usually cheaper than it looks. No key table, no TTL, no extra read — derive the primary key from the client's key and let the conditional create you already have do the work. The expensive part isn't the storage, it's being disciplined about which cancellations you're allowed to interpret as a replay.

One key per intended write, not per attempt. The automatic retries are the obvious duplicate source. The human tapping the button again after a lie is the one that gets missed, and it's the one that produced most of her four posts.

Author's note

What stuck with me was the doc comment above the retry loop. It says the policy retries "on transient failures only — network/timeout errors and 5xx. A 4xx is a definitive answer (e.g. a wrong OTP) and is never retried."

That's careful writing. Somebody thought about it, drew a real distinction, and wrote down why. And it's reasoning about entirely the wrong question — whether the failure is likely to recur, rather than whether the request might already have been processed. Reading it, you nod and move on, because it sounds exactly like someone who considered the problem.

I'd also told the team, earlier and in writing, that no data was being lost — after tracing the happy path end to end and finding it clean. It was clean. I just hadn't looked at the failure path, which is where both of these bugs lived.

The safeguard that's missing announces itself eventually. The one that's present, well-argued, and answering a slightly different question than the one you have is the one that costs you four posts and a week.