Engineering · 2026-10-08

Idempotency Keys for Identity Webhooks: Exactly-Once Trust Decisions

By J. W. Bouckaert

At least once is the only promise anyone makes

Read the delivery guarantees of the systems an identity stack talks to and one phrase recurs. Twilio's Event Streams "guarantees at-least-once delivery of events" and adds that "if you require exactly-once delivery, then you need to implement deduplication in the service that receives the events". Okta's event hooks: "Events are delivered at least once." Auth0's log streams: "We guarantee logs are delivered at least once." Stripe retries a webhook "for up to three days with an exponential back off in live mode" and tells receivers they "might occasionally receive the same event more than once", so they should log the event ids they have processed. The Standard Webhooks specification, written by people who run webhook infrastructure for a living, says the same thing from the sender's side: retry for days, and give the receiver a stable id to deduplicate on.

This is not laziness. Tyler Treat's 2015 essay put the reason in one sentence: "Within the context of a distributed system, you cannot have exactly-once message delivery." The Two Generals problem and the FLP result are impossibility results, and every queue that offers a guarantee at all offers at-least-once. What can be had is exactly-once processing, and Treat named the two ways to get it: "Either the messages themselves should be idempotent, meaning they can be applied more than once without adverse effects, or we remove the need for idempotency through deduplication." Kafka's idempotent producer, which deduplicates on a producer id and a sequence number, is the second approach built into a broker.

For most webhooks a duplicate is a nuisance. For a trust decision it is a second decision. A SIM-swap signal that arrives twice and is processed twice locks an account twice, sends the customer two alerts, opens two cases, and if the playbook charges a step-up verification, charges it twice. The unit of correctness for an identity pipeline is therefore an exactly-once decision: one signal, however many times it is delivered, yields one decision and one set of side effects. Delivery stays at-least-once; the deduplication moves to where the decision is made.

The standard that did not arrive

There was going to be an RFC for this. The IETF HTTPAPI working group's draft on the Idempotency-Key header field, in its seventh revision of 15 October 2025, defines the key as "a unique value generated by the client which the resource uses to recognize subsequent retries of the same request", requires that it "MUST NOT be reused with another request with a different request payload", recommends a UUID, allows the server to fingerprint the payload, and gives the two answers a server should give when something is wrong: a 422 when a key is reused with a different body, and a 409 when a retry arrives while the original is still being processed. A retry of a completed request "SHOULD respond with the result of the previously completed operation, success or an error".

The draft expired on 18 April 2026 and no eighth revision has appeared. That leaves the field as it was: Stripe's contract, which the draft largely codified, and every other API's imitation of it. Stripe's rules are the reference point because they are the most exercised. Keys are up to 255 characters and V4 UUIDs are suggested. The result of the first request is saved "regardless of whether it succeeds or fails", including a 500, and replayed for at least 24 hours with an Idempotent-Replayed header. A key reused with different parameters is an error; a key in concurrent use is a 409. And one line most integrators skip: "Avoid using sensitive data (for example, email addresses or personal identifiers) as idempotency keys."

Brandur Leach's 2017 write-up of the Postgres implementation behind that contract supplies the mechanics: a table with a unique index on the client and the key, the request parameters and the response stored alongside, a lock taken on the row while the request runs, and recovery points between the atomic phases of the operation so that a retry resumes from the last phase that completed rather than repeating the ones that did. Those four ideas, claim before acting, store the result, refuse a changed body, and resume from a recovery point, are the whole design. Every implementation is a choice about how to do each one.

Inbound: a signal sent twice yields one decision

Our ingest endpoint is where a tenant's systems send a signal (a SIM swap observed, a port-out, a credential found in a breach) and where a playbook turns it into a decision and, often, into outbound actions. A client that must not send a signal twice adds one header, and the endpoint's contract is the draft's, with the numbers filled in.

Three details in the order of operations carry the correctness. The key is validated with the body before any round trip, so a malformed key is a 400 that costs nothing. The claim is taken after the caller is authenticated and its signature verified and before any side effect, so an unauthenticated request cannot occupy a key and a request that the client asked to make idempotent never runs unclaimed. And the claim is an insert into a table whose primary key is the tenant and the key digest, written as an insert with on conflict do nothing: two concurrent requests with one key get one row and one refusal, because the database's uniqueness check is the race decider rather than any read-then-write in application code. PostgreSQL's documentation states the property that makes this work, that RETURNING yields only the rows that were inserted, so the winner knows it won from the same statement.

The fourth detail is the recovery point. A signal's processing has two halves: the playbook actions, which may call out to the tenant's systems, and the persistence of the event and its results. If the actions ran and the persistence failed, a naive retry would run the actions again. The claim row therefore moves through three states, in flight, dispatched and completed, and a retry that finds a row in the dispatched state with stored action results skips the actions and finishes the persistence with those results. That is Leach's recovery point applied to the one place in the pipeline where a repeat has a cost outside our own database.

Why hash the key? Because it is a tenant-chosen string of unknown entropy that never needs to be read back, and because Stripe's warning is right: integrators do put identifiers in idempotency keys, and a table of keys that contained phone numbers would be a table of phone numbers. Storing the digest keeps the zero-PII property even against an integrator who ignored the advice.

Outbound: a decision delivered twice yields one action

The other direction is harder, because the receiver is code we do not write. When a playbook fires a callback into a tenant's fraud system, the delivery is at-least-once like everyone else's: one attempt inside the ingest request with a five-second deadline, then, on a 5xx, a 429, a network error or that deadline, a queue and up to three more attempts from a worker that runs on a five-minute cron. Any other 4xx is terminal, on the reasoning we learned running a CAEP transmitter: a 4xx is a verdict, a 5xx is a retry, and re-sending a request the receiver called malformed will not improve it.

Senders differ in what they give the receiver to deduplicate on. GitHub's X-GitHub-Delivery is a GUID that "will be the same as in the original delivery" on a redelivery. The Standard Webhooks id "remains the same no matter how many times a webhook that has failed is retried", while its timestamp is updated per attempt and the signature covers id, timestamp and body together. Stripe generates "a new signature and timestamp for the new delivery attempt" and expects the receiver to key on the event id in the body. Our callback envelope carries no delivery id header. What it carries is a timestamp that is stamped once, when the envelope is built, and repeated unchanged on every retry, so the tuple of tenant, event type, hashed identifier and timestamp is stable across the retries of one delivery and distinct between two deliveries at millisecond resolution. A receiver keys on that tuple, or, better, puts its own unique field in the ingest payload and reads it back out of the envelope's metadata, which does not depend on our timestamp resolution at all. The integration post has the code.

What a receiver cannot rely on is byte identity, since a retry is re-serialised from the queue row and key order may differ. So the receiver verifies the HMAC-SHA-256 signature against the exact bytes of the request in front of it, then parses, then builds the deduplication key from parsed fields. Signature first, dedup second: a duplicate is discarded only after it has proved it came from us, and Twilio's advice applies, return a 200 for the duplicate you discard, because a 5xx would earn you another copy.

Windows, tolerances and the two clocks

Every deduplication scheme has a horizon, and the horizon has to be longer than the sender's retry schedule or the scheme has a hole exactly where retries land. Stripe prunes keys after 24 hours and retries webhooks for three days; the two are different features with different windows, and a receiver that remembered Stripe event ids for only a day would reprocess a delivery retried on day two. Standard Webhooks' consumer guidance suggests saving ids for as little as five minutes, which is sound only for a sender whose retry schedule is that short. Our inbound keys live 24 hours; our outbound retries finish within the worker's schedule, so a receiver that remembers our deduplication tuple for a day is safe by a wide margin.

Replay protection is a different control that is easy to confuse with idempotency. A signature timestamp tolerance, five minutes by default at Stripe and in Svix's libraries, rejects a captured request that is replayed later; it says nothing about a legitimate retry inside the window, which is exactly when duplicates arrive. Idempotency handles the second case. A receiver needs both, and needs to notice which clock each one uses: the signature timestamp is refreshed per attempt by most senders and so measures freshness, while a deduplication id is fixed at creation and so measures identity. Our envelope's fixed timestamp is a deduplication field; the replay check on our inbound path is a separate header with its own tolerance. Reading one as the other is how a receiver ends up rejecting honest retries as replays, or accepting replays as retries.

A receiver's checklist

For the team wiring a fraud system, a rules engine or a case manager to any identity webhook, ours included:

1. Send an idempotency key on every signal you post, a UUID per signal, and never an identifier. Treat a 422 as a bug in your key generation and a 409 as an instruction to wait two seconds.
2. On receipt, verify the signature over the raw bytes before you parse anything, and treat an absent signature as unauthenticated unless you have deliberately chosen not to configure one.
3. Deduplicate on a field that is stable across retries: the sender's delivery id where there is one, our envelope tuple or your own echoed field where there is not. Keep the dedup store for longer than the sender's retry horizon.
4. Make the action itself idempotent where you can. A lock that is already held is a no-op; a case that already exists is found rather than opened. This is Treat's first strategy and it makes the second one a safety net rather than a load-bearing wall.
5. Answer fast and work afterwards. Accept, record the dedup key and the raw body in one transaction, return 2xx, then process. A slow receiver turns an inline delivery into a queued one and receives its decision minutes later.
6. Return 2xx for a discarded duplicate. A 5xx asks for another copy.

Left open

Two things we do not do, stated plainly. Our outbound envelope has no delivery id header, and Standard Webhooks' three-header convention would be the obvious thing to adopt; the envelope tuple works, but a header is easier to explain and harder to get wrong, and it is on the list. And idempotency keys on the ingest path are optional, because a great many integrators retry without them and refusing those requests would be the wrong default; the cost is that a client without a key has at-least-once semantics on the signal path too, and the playbook actions have to tolerate it. The SIM-swap detection flow was designed with that in mind: a deny that has already been issued is issued once, whatever the delivery count. Whether the IETF draft is revived, and whether it settles on the 422 and 409 the field already uses, will decide only what the header is called.

Start free · Test the API