Testing Webhook Retries and Idempotency Contracts

Symptom: a customer was refunded twice, and the logs show two deliveries of the same event four minutes apart — the second because the first response took eleven seconds and the sender timed out at ten. The delivery worked exactly as documented. The receiver was not idempotent. This guide is part of Webhook and Callback Contracts and covers making at-least-once delivery safe, and proving it.

Root Cause: At-Least-Once Is a Guarantee, Not a Warning

Every retrying delivery system is at-least-once, because the sender cannot distinguish “the receiver never got it” from “the receiver got it and the acknowledgement was lost”. A duplicate is not a malfunction; it is the specified behaviour. The receiver is the only place the problem can be solved.

How a successful delivery becomes a duplicate The first delivery is processed successfully but the response arrives after the sender's timeout, so the sender records a failure and retries; the receiver then processes the same event a second time. sender receiver POST evt_9 — first attempt refund issued — 11 seconds 200 OK — but the sender gave up at 10s POST evt_9 — retry, same event id refund issued AGAIN

The picture also shows the cheapest mitigation: acknowledge in milliseconds and do the work afterwards. That removes most duplicates but not all, so idempotency is still required.

Step 1: Make the Effect Idempotent in the Database

Application-level checks — “have I seen this id?” then “do the work” — fail under concurrency, which is exactly the situation a retry storm produces. Put the guarantee where concurrency is already handled.

-- One row per handled event. The primary key IS the idempotency mechanism.
CREATE TABLE processed_events (
  event_id     TEXT PRIMARY KEY,
  event_type   TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Retention: keep ids for twice the sender's full retry window.
CREATE INDEX processed_events_processed_at_idx ON processed_events (processed_at);
// webhooks/process.ts
export async function processEvent(event: WebhookEnvelope): Promise<"applied" | "duplicate"> {
  return db.transaction(async (tx) => {
    try {
      // Claim the event FIRST, in the same transaction as the side effect.
      await tx.query(
        "INSERT INTO processed_events (event_id, event_type) VALUES ($1, $2)",
        [event.id, event.type],
      );
    } catch (err) {
      if ((err as { code?: string }).code === "23505") return "duplicate"; // unique_violation
      throw err;
    }

    // If this throws, the transaction rolls back and the claim disappears,
    // so the next retry is free to try again. That is the behaviour we want.
    await applySideEffect(tx, event);
    return "applied";
  });
}

Why this works: the claim and the effect commit together. Two concurrent deliveries race on the primary key, exactly one wins, and the loser learns it lost before doing anything.

Step 2: Separate Duplicates From Stale Updates

Deduplication answers “have I seen this event?”. It does not answer “is this event still relevant?”. Two different events about one resource, delivered out of order, both pass the duplicate check.

// Applied inside the transaction, after the claim succeeds.
async function applySideEffect(tx: Tx, event: WebhookEnvelope) {
  if (event.type === "order.status_changed") {
    // Discard an update older than the state we already hold.
    const { rowCount } = await tx.query(
      `UPDATE orders
          SET status = $1, status_version = $2
        WHERE id = $3 AND status_version < $2`,
      [event.data.status, event.data.version, event.data.order_id],
    );
    if (rowCount === 0) {
      logger.info({ msg: "stale event discarded", id: event.id, version: event.data.version });
    }
  }
}

The status_version < $2 predicate is the ordering guard. It makes the update a no-op when a newer version has already been applied, which is the correct behaviour for out-of-order delivery and costs nothing when events arrive in order.

Two guards, two different problems The event id guard rejects the same event arriving twice; the version guard rejects a different, older event arriving after a newer one. Neither substitutes for the other. evt_9 (again) same id, same content evt_7 (late) different id, older version unique index on event_id catches it — no effect applied index does NOT catch it the id has never been seen version predicate update affects zero rows both guards run, in the same transaction

Step 3: Test the Duplicate, the Race and the Reorder

Three tests, and the second one is the one usually missing.

// webhooks/idempotency.test.ts
const event = {
  id: "evt_9tX2mLq7Rk3vBn5cWd8pYs1a",
  type: "refund.succeeded" as const,
  created_at: "2026-07-29T10:00:00Z",
  api_version: "2026-07-01",
  data: { order_id: "ord_42", amount: 4200, currency: "EUR" },
};

test("a repeated delivery applies the effect once", async () => {
  expect(await processEvent(event)).toBe("applied");
  expect(await processEvent(event)).toBe("duplicate");
  const { rows } = await db.query("select count(*) from refunds where source_event = $1", [event.id]);
  expect(Number(rows[0].count)).toBe(1);
});

test("concurrent deliveries of the same event apply the effect once", async () => {
  // The case an application-level "have I seen this?" check gets wrong.
  const results = await Promise.all([processEvent(event), processEvent(event), processEvent(event)]);
  expect(results.filter((r) => r === "applied")).toHaveLength(1);
  expect(results.filter((r) => r === "duplicate")).toHaveLength(2);
});

test("an out-of-order status update is discarded", async () => {
  const newer = statusEvent({ id: "evt_b", version: 5, status: "delivered" });
  const older = statusEvent({ id: "evt_a", version: 4, status: "shipped" });
  await processEvent(newer);
  await processEvent(older);                       // arrives late
  const { rows } = await db.query("select status, status_version from orders where id = $1", ["ord_42"]);
  expect(rows[0]).toEqual({ status: "delivered", status_version: 5 });
});

Step 4: Test the Retry Storm

After an outage, every queued delivery arrives at once. The behaviour to prove is that the receiver stays correct — throughput can degrade, correctness cannot.

What a retry storm must produce Two hundred deliveries arrive at once after an outage, half of them duplicates and in random order; the receiver may slow down, but exactly one hundred effects are applied. 200 deliveries at once 100 events, shuffled, each twice unique index on event_id claim, then apply exactly 100 effects 100 duplicates, no-ops Throughput may degrade under a storm; correctness may not. 
test("a burst of 200 deliveries, half duplicates, applies each effect once", async () => {
  const events = Array.from({ length: 100 }, (_, i) => refundEvent(`evt_${i}`));
  const burst = [...events, ...events].sort(() => 0.5 - Math.random());  // shuffled duplicates

  const results = await Promise.all(burst.map((e) => processEvent(e)));
  expect(results.filter((r) => r === "applied")).toHaveLength(100);

  const { rows } = await db.query("select count(*) from refunds");
  expect(Number(rows[0].count)).toBe(100);
});

Verification

$ npm test -- webhooks/idempotency.test.ts

 PASS  webhooks/idempotency.test.ts
  ✓ a repeated delivery applies the effect once (24 ms)
  ✓ concurrent deliveries of the same event apply the effect once (31 ms)
  ✓ an out-of-order status update is discarded (18 ms)
  ✓ a burst of 200 deliveries, half duplicates, applies each effect once (412 ms)

Tests: 4 passed, 4 total

Against a running receiver, the end-to-end proof is a replayed delivery with no second effect:

for i in 1 2 3; do
  curl -s -o /dev/null -X POST http://localhost:3000/webhooks \
    -H "X-Signature: $SIG" -H "X-Signature-Timestamp: $TS" \
    --data-binary @fixtures/refund-succeeded.json
done
psql -tAc "select count(*) from refunds where source_event = 'evt_9tX2mLq7Rk3vBn5cWd8pYs1a';"
# 1

Edge Cases and Caveats

  • Side effects outside the database. Sending an email or calling a payment provider cannot be rolled back with the transaction. Claim the event, commit, then perform the external call, and make the external call itself idempotent using the event id as its idempotency key.
  • Retention that is too short. Deleting processed ids after an hour when the sender retries for a day reintroduces duplicates in exactly the situation the mechanism exists for. Twice the retry window is the safe default.
  • Senders that do not guarantee a stable id. Raise it as a contract defect. Hashing the payload is the fallback, and it breaks the day the sender adds a field — so document the workaround as temporary and revisit it.
  • Deduplicating in the queue rather than the handler. Tempting, but a queue’s deduplication window is usually minutes, not the sender’s full retry schedule. Keep the authoritative check next to the side effect.
  • Multiple receiver instances. The unique index handles this correctly, which is precisely why the guarantee belongs in the database rather than in process memory. An in-memory set of seen ids is wrong the moment you run two replicas.

What to Do With Events That Never Succeed

A retry schedule ends. The events that exhaust it are the ones most likely to matter, and a receiver needs a documented answer for them.

Three arrangements are common. Abandonment with notification is the simplest: the sender stops trying and tells the endpoint owner, leaving recovery to a manual replay. A dead-letter queue on the sender’s side keeps the payload retrievable for a documented window, which turns recovery into an API call. Receiver-side reconciliation inverts the responsibility: the receiver periodically fetches the current state of anything it cares about, so a lost event is corrected within one reconciliation cycle rather than lost.

The third is the most robust and the least discussed. Webhooks are an optimisation — a way to learn about a change sooner than polling would. A receiver that also reconciles is resilient to the sender’s worst day; a receiver that treats delivery as the only source of truth is not. Where the data matters, say so in the contract: publish the endpoint that lists current state, and recommend a reconciliation interval.

Frequently Asked Questions

What should the deduplication key be?

The sender’s event id, provided the sender guarantees it is stable across retries. A payload hash breaks when the sender adds a field; a resource id collapses distinct events on the same resource. If the sender does not guarantee a stable id, that is a contract defect worth raising before writing any receiver code.

Is a unique index enough to make a receiver idempotent?

It is the strongest single mechanism, because it holds under concurrency where an application-level check does not. Insert the event id first inside the same transaction as the side effect, and treat a duplicate-key violation as a successful no-op.

How long should processed event ids be kept?

At least as long as the sender’s full retry schedule, plus a margin — typically twice the window. If the sender abandons after 24 hours, keeping ids for 72 hours covers every retry with room for a clock discrepancy.

What if two events for the same resource arrive out of order?

Deduplication does not solve ordering. Carry a monotonic version or an occurrence timestamp in the payload and discard an update older than the state you already applied, which is a separate check from the duplicate check.

Should the receiver return 200 for a duplicate?

Yes. The delivery was handled — the fact that it was handled earlier is not the sender’s problem, and returning an error would trigger a retry of something already done.

Related Articles