The request succeeded. The database committed, the card was charged, the row was written — and then the response never made it back. A load balancer timed out, a mobile connection dropped, a pod got evicted mid-flush. The client has no way to tell "my request failed" from "my request worked but the answer got lost," so it does the only sane thing: it retries. And now you've charged the card twice.
The takeaway up front: a network is not a transaction, so any request that changes state will eventually be sent more than once. The fix is not to stop clients retrying — retries are correct and necessary. The fix is to make the second identical request a no-op that returns the first one's result. That property is idempotency, and the standard way to bolt it onto an otherwise unsafe endpoint is an idempotency key: a unique token the client generates, sends with the request, and reuses on every retry of that same logical operation.
Why retries are unavoidable, not a bug
It's tempting to treat duplicate requests as a client mistake to be scolded away. They aren't. In any distributed call there is a window between "the server finished the work" and "the client received confirmation," and nothing the client can observe distinguishes a failure in that window from a failure before the work ran. A robust client must retry on timeout — giving up would drop real, successful-but-unconfirmed operations on the floor.
Some HTTP methods are already safe under retry. GET, PUT, and DELETE are defined to be idempotent: fetching a resource twice, or setting it to a specific value twice, or deleting it twice, all leave the system in the same state as doing it once. The problem child is POST, which means "create a new thing" or "run this action." Two identical POST /charges requests are supposed to create two charges. That's exactly the semantic you don't want when the second one is a panicked retry of the first.
You cannot fix this by making the client smarter about when to retry. The information it needs — did the first attempt actually land? — lives on the server. So the server has to be the one to recognize the duplicate.
What an idempotency key actually is
An idempotency key is a client-generated unique string, typically a UUID, that names one logical operation. The client sends it in a header:
POST /charges HTTP/1.1
Content-Type: application/json
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
{"amount": 4200, "currency": "usd", "source": "card_YOUR_TOKEN"}
The contract is simple and it belongs to the client: generate the key once, before the first attempt, and send the same key on every retry of that operation. Generate a new key only when you start a genuinely new operation. If the client makes a fresh UUID on each retry, the mechanism is useless — every attempt looks new to the server.
On the server side, the key becomes a lookup. The first time you see a key, you do the work and record the result under it. Every subsequent request carrying that key returns the recorded result without repeating the work. The client can't tell whether its request was the one that did the work or the one that got a replay — and it doesn't need to. Both got the same answer.
The server-side flow, step by step
A correct implementation is a small state machine keyed on (account, idempotency_key). Scoping to the account matters — see below — but the core loop is:
- Read the request's idempotency key. If it's missing, either reject the request or process it without protection; be explicit about which.
- Look up the key. If a completed record exists, return its stored status code and body verbatim. Done — no work happens.
- If no record exists, atomically insert one in a pending state. Winning that insert is your lock: it means you own this operation.
- Do the real work — charge the card, insert the row — inside a transaction.
- Save the response (status + body) against the key, mark it completed, and return it.
The subtle part is step 3. Two retries can arrive nearly simultaneously — a client fired a second attempt just as the first was landing — and a naive "check, then insert" has a race between the check and the insert where both requests see "no record" and both proceed. That's the exact double-charge you were trying to prevent, now hiding inside your fix.
Making it race-proof
The reliable way to close that window is to let the database enforce uniqueness, not your application code. Put a unique constraint on the key column and treat a duplicate-key insert failure as a signal rather than an error:
CREATE TABLE idempotency_keys (
id bigserial PRIMARY KEY,
account_id bigint NOT NULL,
idem_key text NOT NULL,
status text NOT NULL DEFAULT 'pending', -- pending | completed
response_code int,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (account_id, idem_key)
);
Now the insert is the lock. Exactly one concurrent request wins the INSERT; the others get a unique-violation and know a sibling is already handling this key:
def process(account_id, idem_key, do_work):
try:
with db.transaction():
db.execute(
"INSERT INTO idempotency_keys (account_id, idem_key) VALUES (%s, %s)",
(account_id, idem_key),
)
except UniqueViolation:
# Someone else owns this key. Return the finished result, or tell the
# client to retry if that sibling is still in flight.
row = db.fetchone(
"SELECT status, response_code, response_body FROM idempotency_keys "
"WHERE account_id=%s AND idem_key=%s", (account_id, idem_key))
if row.status == "completed":
return row.response_code, row.response_body
return 409, {"error": "request already in progress, retry shortly"}
# We won the insert — we own this operation. Do the work exactly once.
code, body = do_work()
db.execute(
"UPDATE idempotency_keys SET status='completed', response_code=%s, "
"response_body=%s WHERE account_id=%s AND idem_key=%s",
(code, Json(body), account_id, idem_key))
return code, body
Two details make or break this. First, wrap the real work and the "mark completed" write so the record can never say completed while the side effect rolled back, or vice versa — ideally the same transaction, and if the side effect is an external call (a payment processor), lean on its idempotency key too so the whole chain is covered. Second, return 409 Conflict (or 425 Too Early) when a sibling is still pending, rather than doing the work or blocking indefinitely; the client retries, and by then the winner has usually finished and the retry gets the stored 200.
The rules that keep it honest
Scope keys to the caller. Store and match on (account_id, idempotency_key), never the key alone. Clients generate UUIDs independently; if two customers ever collide on a value in a global namespace, one would receive the other's response — a data leak. Scoping to the account makes collisions harmless.
Bind the key to the request body. A key promises "this exact operation." If a client reuses a key with a different payload — a real bug on their side — replaying the old response silently hides it. Store a hash of the request body with the key and return 422 if the same key arrives with a different body, so the mistake surfaces loudly instead of corrupting data.
Expire keys deliberately. Retries happen within seconds to minutes, not weeks, so records need only a short life. Keep them 24 hours to a few days, then sweep them; a small table stays fast, and you avoid it degrading into the kind of slow, contended hotspot that eventually shows up as connection pool exhaustion.
Only guard what mutates. Idempotency keys are for POST and other unsafe operations. GET needs no key. This is one piece of a broader stance on predictable endpoints — status codes, versioning, error shapes — covered in the REST API design guide.
Quick reference
| Decision | Do this | Because |
|---|---|---|
| Who generates the key | The client, once per operation | Only the client knows two attempts are the "same" request |
| Which endpoints need it | POST and other unsafe writes |
GET/PUT/DELETE are already idempotent |
| How to prevent races | Unique constraint on (account, key) |
The DB insert becomes the lock; app-level checks race |
| Concurrent duplicate, still pending | Return 409/425, let client retry |
Avoids doing the work twice or blocking |
| Same key, different body | Reject with 422 |
A reused key on new data is a client bug, not a retry |
| Key lifetime | Expire after hours-to-days | Retries are short-lived; old rows only add cost |
FAQ
What is an idempotency key?
It's a unique token, usually a UUID, that a client generates and attaches to a state-changing request so the server can recognize retries of that exact operation. The first request does the work and stores its result under the key; any later request with the same key returns the stored result instead of repeating the work.
When should I use idempotency keys?
On any endpoint where performing the operation twice causes harm and where clients may retry — creating charges, placing orders, sending messages, provisioning resources. In practice that means non-idempotent methods like POST. Safe methods (GET, PUT, DELETE) already produce the same state under repetition and don't need a key.
Do idempotency keys guarantee exactly-once execution?
They give you effectively-once observable behavior, which is the practical goal. True exactly-once delivery is impossible over an unreliable network, but by doing the work once and replaying the stored response on every duplicate, the client always sees a single, consistent result — which is what "exactly once" needs to mean in practice.
Where should the client put the idempotency key?
By widespread convention, in an Idempotency-Key HTTP header carrying a UUID. A header keeps it out of the request body and URL, so it's easy to log, inspect, and handle in middleware without parsing the payload. Document the exact header name in your API reference so clients don't guess.
How long should the server keep idempotency keys?
Long enough to cover any realistic retry window and no longer — typically 24 hours to a few days. Retries after a failed request happen within seconds or minutes, so a short retention covers real cases while keeping the lookup table small and fast. Sweep expired rows on a schedule.
What happens if two identical requests arrive at the same time?
Without protection, both can pass a naive existence check and execute — the double-write you were preventing. The fix is a unique database constraint on the scoped key so exactly one insert wins and becomes the lock; the losers detect the conflict and either return the finished response or a 409 telling the client to retry once the winner completes.
Next step
Pick the single most dangerous endpoint you run — the one that charges a card, creates an order, or sends money — and add idempotency to it before you add it anywhere else. Create the keys table with a unique constraint on (account, key), accept an Idempotency-Key header, and make a repeat key return the stored response instead of redoing the work. One table and one header turn "the client retried and we double-charged them" into a non-event. More pragmatic engineering guides are at TheAppCode.
