TL;DR
MessageDeduplicationIdsounds like an identifier for the message. It isn't — it identifies the operation the message represents. Two deliveries of one webhook are two messages and one operation.- Any key computed from when you saw it rather than what it is cannot recognise a repeat. If your key would differ on a replay, it isn't a deduplication key.
- SQS FIFO remembers a deduplication ID for five minutes and the interval is fixed. That covers bursts, not a sender backing off over hours. The queue is a first line, never the mechanism.
- At-least-once delivery is the normal operating mode of everything in the chain — webhook senders, SQS, SNS, async Lambda invocations, load balancers, users clicking twice. The design question is never "will this arrive twice", it's "what happens when it does".
- Prefer the sender's natural identifier. It's stable across retries and lets you correlate with their logs.
- Use a conditional write, not read-then-write. Retry storms are exactly when your race condition is concurrent.
- Claim-then-fail silently drops work. Release the key on failure, or record status against it.
- Your status code is part of your retry protocol. Returning 200 means "I've taken responsibility for this." If you haven't, return 500.
I was brought in to review an integration layer — a webhook receiver forwarding inbound events onto a FIFO queue — and stopped on two adjacent lines:
await sqs.send(new SendMessageCommand({
QueueUrl: queueUrl,
MessageBody: JSON.stringify(payload),
MessageGroupId: event.id,
MessageDeduplicationId: `${Date.now()}-${Math.random()}`
}));The first of those is right. Message group ID controls ordering — messages sharing a group are delivered in order, different groups process in parallel. Using the sender's own event identifier gives per-event ordering with full parallelism across events. Someone thought about that.
The second takes the one field SQS FIFO gives you for free to make redelivery safe and fills it with a value guaranteed to be unique on every call. Two identical deliveries produce two different deduplication IDs, so the queue treats them as distinct messages and delivers both.
The natural, stable key is on the line above.
I've kept coming back to it since, because it isn't ignorance. Whoever wrote it knew what FIFO queues are for and reached for the right identifier once. Then they defeated deduplication with something that looks, at a glance, like careful work. Reading it in a diff, it's a unique ID being generated. It passes.
The fix is to use the identifier that's already there:
await sqs.send(new SendMessageCommand({
QueueUrl: queueUrl,
MessageBody: JSON.stringify(payload),
MessageGroupId: event.id,
MessageDeduplicationId: event.id // the operation, not the delivery
}));The same value in both fields is fine. They're independent parameters answering different questions: MessageGroupId asks what must stay in order relative to this?, MessageDeduplicationId asks have I already accepted this? For a stream of per-event webhooks, the sender's event ID is the honest answer to both.
That one change buys less than it appears to, and the gap matters. SQS remembers a deduplication ID for five minutes, and the interval is not configurable. Inside the window a duplicate is accepted by the API and silently not delivered — the sender sees success, the consumer never sees the message, which is exactly the behaviour you want. At six minutes it's a new message.
So the five-minute ceiling decides how the problem gets divided. Bursts live inside it: a load balancer retry, an immediate sender retry, a user clicking twice. Webhook senders backing off over hours do not. The queue handles the thundering herd; everything slower is your handler's problem, which is what the rest of this is about. Treat the dedup ID as a cheap first line, never as the mechanism.
Why the mistake is structural
There's a specific trap here, and "remember to set the dedup ID properly" is useless advice against it.
MessageDeduplicationId sounds like an identifier for the message. Identifiers should be unique. Generating a unique value is what you do with an ID field. The name leads you directly to the wrong answer.
But it isn't an identifier for the message. It identifies the operation the message represents — it answers "is this the same thing I already accepted?" Two deliveries of one webhook are two messages and one operation. The field wants the operation's identity.
Worth being precise about where that criticism lands: it's aimed at the API, not at anything you can change in your own code. The field is called MessageDeduplicationId and it will stay that way. Had AWS named it OperationKey or IdempotencyKey, the mistake would be close to impossible — the name would ask the right question at the point of use. In distributed primitives the vocabulary is part of the safety surface, and this one walks competent engineers into the wrong answer.
The same confusion produces the same bug in other shapes. Idempotency keys generated inside the handler rather than derived from the request. Deduplication windows keyed on receipt time. Anything computed from when you saw it rather than what it is has the identical defect: it cannot recognise a repeat, because a repeat happens at a different time.
A deduplication key must be derivable identically by anyone holding the same logical request, at any time. If your key would differ on a replay, it isn't one.
Everything redelivers
This matters because at-least-once isn't an edge case. It's the normal operating mode of every link in the chain:
- Webhook senders retry on timeout, on 5xx, and on a connection dropped after your handler committed. They cannot distinguish "didn't arrive" from "arrived and the acknowledgement was lost".
- SQS delivers at least once by design. Visibility timeout expiry redelivers.
- SNS retries failed Lambda deliveries.
- Lambda retries asynchronous invocations twice by default.
- Load balancers and gateways retry.
- Users retry, by clicking twice.
None of these are faults. They're the correct behaviour of systems that cannot tell those two cases apart. The Two Generals problem isn't solvable, so everything above chooses duplication over loss — which is the right choice, and it hands the problem to you.
Where it hurts most
The endpoints that most need replay protection are usually the ones that have none: payment notifications, case creation, anything that writes to a system of record.
The queue call was the visible half. The handler behind it is where the real damage lives, and this is the shape I expect to find in an integration like that one — not a quotation from it, but the pattern I keep meeting:
export const handler = async (event) => {
const payload = parse(event.body);
try {
if (payload.result === SUCCESS_CODE) {
await createRecordInCoreSystem(payload);
}
await db.query(`UPDATE payments SET status = $1 WHERE reference = $2`, [...]);
} catch (err) {
console.error(err);
}
return { statusCode: 200, body: JSON.stringify({ success: true }) };
};Two failures, and the second is worse than the first.
No deduplication. A redelivery creates a second record for one payment. Frequently the sender's transaction reference is right there in the payload — used as free text in a description field, visible to a human, invisible to any logic. The natural key present and used as decoration.
Unconditional success. The catch swallows both the downstream call and the database write, then returns 200. So when this fails, the sender is told the notification was accepted and will not retry. The mechanism that exists specifically to recover from this failure has been switched off by the response code, and the only trace is a log line.
That's the mirror image of the idempotency problem. Missing idempotency means retries cause damage. Unconditional success means retries never happen. A system with both has no delivery guarantee in either direction — it duplicates when it succeeds twice and drops when it fails once.
Your status code is part of your retry protocol. Returning 200 to a sender means "I have taken responsibility for this." If you haven't, return 500 and let them try again. The only honest reason to acknowledge a failure is when you've durably captured the payload for later processing — and then the 200 is true, because you really did take responsibility.
Four decisions
None of these are hard. All of them need making explicitly.
1. What is the key? Prefer the sender's natural identifier — webhook event ID, payment reference, message ID. Stable across retries, meaningful in a support conversation, and it lets you correlate against the sender's own logs. Only hash the payload if there genuinely isn't one, and know that some senders vary payloads between retries, which breaks payload hashing.
2. Where does the record live? A table with the key as primary key, or a DynamoDB item. The critical property is that the uniqueness check and the work cannot interleave — which means a conditional write, not read-then-write:
INSERT INTO processed_events (event_key, status, received_at)
VALUES ($1, 'in_progress', now())
ON CONFLICT (event_key) DO NOTHING
RETURNING event_key;or, in DynamoDB:
await ddb.send(new PutItemCommand({
TableName: 'processed_events',
Item: { event_key: { S: key }, status: { S: 'in_progress' } },
ConditionExpression: 'attribute_not_exists(event_key)'
}));No row returned — or a ConditionalCheckFailedException — means someone else has it. One round trip, atomic.
A SELECT followed by an INSERT is a race, and it's a race you'll lose specifically under retry storms, because retries arrive in bursts and bursts are exactly when they're concurrent.
3. How long do you remember? Long enough to cover the sender's retry window with margin. Most providers retry with backoff over hours; a day is comfortable, a week is safer. It's a TTL and it's cheap — DynamoDB will expire it for you.
4. What does a duplicate return? Ideally the original response, so the caller sees a consistent answer. At minimum, success. A duplicate is not an error, and returning 409 will make well-behaved senders retry harder.
flowchart TD
IN[Inbound event] --> V{signature valid?}
V -->|no| R1[401]
V -->|yes| K[derive key from<br/>sender's identifier]
K --> C{conditional write}
C -->|conflict, status complete| DUP[200 — already processed]
C -->|written| W[do the work]
W -->|ok| M[mark complete] --> OK[200]
W -->|fail| REL[release or mark failed]
REL --> ERR[500 — let them retry]The step everyone misses
Look at the release or mark failed branch. That's the one people leave out, and leaving it out is worse than having no idempotency at all.
If you claim the key, then the work fails, then you return 500 — the retry arrives, sees a conflict, and skips. The operation is silently lost. You've built a system that reliably drops work, and the mechanism is the thing you added to make it safe.
Two ways to avoid it. Release the key on failure, so the retry can proceed. Or store a status against it (in_progress / complete) and let a retry reprocess anything not marked complete — which is more robust, because it also handles the case where your function is killed before it can release anything.
Claim-then-fail is how "idempotent" handlers quietly drop work. If you take one thing from this article, make it that one.
The nuance
I don't want to leave this as "add idempotency everywhere", because blanket policies are how this machinery gets a reputation for overhead.
Some operations don't need it. Idempotent-by-nature writes — setting a field to a fixed value, upserting complete state — are safe to repeat. A duplicate "set status to CONFIRMED" changes nothing. Wrapping those in a deduplication table is ceremony.
Reads never need it.
"The same request" is a business question, not a technical one. Two identical payment notifications for one transaction: duplicate. Two identical "add note to case" requests, deliberately sent twice by a user: not duplicates. Your key encodes that answer — and getting it wrong in the safe-looking direction, deduplicating things meant to be distinct, silently loses data. That's harder to detect than duplication, because nothing anomalous appears anywhere.
Where at-most-once genuinely is acceptable — write it down. Analytics events, non-critical notifications. That's a legitimate position. It's only a problem when it's undeclared, because an undeclared decision is indistinguishable from an oversight.
The audit
Take an hour. List every place work enters your system from outside — HTTP endpoints, queue consumers, topic subscribers, async invocations, overlapping schedules. For each, answer three questions:
- What happens if this arrives twice? Not "will it" — assume it will.
- What key would identify a repeat? If you can't name one, you don't have idempotency, whatever the code implies.
- What do we return when we fail? If the answer is a success code, the sender's retry mechanism is disabled and you own the recovery.
The value isn't the fixes. It's that a distributed system's correctness is mostly determined by those answers, and in most codebases they have never been written down — which means nobody knows them, including the people who wrote the code.
And go and look at your deduplication IDs. If one of them contains a timestamp or a random number, it isn't a deduplication ID. It's a unique ID sitting in a field that needed exactly the opposite.