Webhook Examples: Build a Receiver That Survives Retries and Crashes

Write the provider contract and durability boundary first

A callback URL is only one part of the system. Before implementation, write down the provider-specific delivery contract and the business effect that must remain correct after repetition.

• Current provider documentation: GitHub explains its exact raw-body signature contract, while Stripe documents a different signed-payload and timestamp contract. Use these as evidence that provider contracts differ, not as interchangeable code.
• A public HTTPS receiver that exposes raw bytes: Your framework must let server code read the exact bytes required by the signature scheme before any JSON middleware, body normalization, character conversion, or field coercion changes them. Bound the request size before buffering it.
• A transactional durable store: Use a database capable of inserting one provider delivery identity under a unique constraint and inserting its pending outbox work in the same transaction. Configure real durability, backups, restore, and isolation separately.
• A worker with a lease and retry budget: The worker must claim pending work, expose lease expiry, use bounded backoff, distinguish terminal from retryable failures, and leave an inspectable state when the dependency remains unavailable.
• A downstream idempotency or reconciliation decision: An outbox makes work durable, but it cannot make an external effect exactly once. Use a provider-supported idempotency key where available, store the returned effect identity, and define how an operator reconciles an ambiguous timeout.

Credentials and access

Choose where acknowledgement ends and business work begins

The central decision is not which framework receives HTTP. It is which state must be durable before the provider sees an acknowledgement and which work is allowed to continue asynchronously.

Recommended:Use one database transaction for the provider delivery identity and pending outbox work, then acknowledge with the exact response the current provider expects. Let a worker lease and retry the outbox. This is more operational work, but it makes the crash boundaries visible instead of depending on a provider retry after a successful acknowledgement.

Build and verify a durable webhook receiver

Define the provider contract, authenticate exact raw bytes, claim event identity and pending work atomically, dispatch with leases and idempotency, and verify crash and recovery behavior.

const rawBody = await readBoundedRawBody(request)
const verified = verifyCurrentProviderSignature({
  rawBody,
  headers: request.headers,
  secret: env.WEBHOOK_SIGNING_SECRET,
})

if (!verified.ok) return providerInvalidSignatureResponse()
if (!currentProviderTimestampPolicyAllows(verified.timestamp)) {
  return providerStaleSignatureResponse()
}

const event = parseAndValidateAllowedEvent(rawBody)

const receipt = await database.transaction(async (tx) => {
  const existing = await tx.webhookEvent.findByProviderId(event.id)
  if (existing) return compareExistingReceipt(existing, sha256(rawBody))

  const stored = await tx.webhookEvent.insert({
    providerId: event.id,
    payloadHash: sha256(rawBody),
    status: 'pending',
  })
  await tx.outbox.insert({
    id: `outbox:${stored.id}`,
    eventId: stored.id,
    status: 'pending',
  })
  return { kind: 'accepted', stored }
})

return responseForCurrentProvider(receipt.kind)

const claim = await outbox.leaseNext({ leaseForMs: 30_000 })
if (!claim) return

try {
  const effect = await downstream.apply({
    idempotencyKey: claim.id,
    event: claim.event,
  })
  await outbox.completeIfLeaseOwned(claim, effect.id)
} catch (error) {
  await outbox.retryIfLeaseOwned(claim, boundedErrorCode(error))
}

Test authentication, identity, crashes, retries, and the live path

The receiver is only trustworthy when failures at every boundary leave a state that can be explained and recovered without guessing.

Diagnose webhook failures from transport to effect

Start with the provider delivery record and follow one stable identity through authentication, durable acceptance, leased processing, and downstream reconciliation.

Operate receipt, work, and external effects as separate states

Deploy

Deploy schema support before receiver code and keep old workers compatible with new pending rows during a rolling release. Store signing and downstream credentials separately, verify direct HTTPS and raw-body access, then register the exact endpoint and allowlisted events with the provider.

The same-release ZIP is locally reproduced evidence only. Before dev or production advances, require its exact public path, the article, the hero asset, the receiver route, and a provider-controlled event to return or behave as expected in that environment.

Monitor

Measure authenticated, rejected, duplicate, quarantined, pending, leased, retrying, completed, and terminal events separately. Alert on oldest pending age, expired leases, repeated signature failures, changed-content conflicts, acknowledgement latency, and reconciliation age.

Correlate with provider delivery ID, internal event ID, outbox ID, and downstream effect ID. Keep signing material, authorization headers, raw private payloads, customer records, and unbounded provider errors out of routine logs.

Recover

For a dependency outage, repair the dependency and resume only eligible pending rows with the same downstream identity. For an ambiguous timeout, reconcile the provider effect before retrying. For secret exposure, rotate through the provider-supported overlap and verify both rejection and legitimate redelivery behavior.

For database or whole-app restore, compare the checkpoint with provider delivery history and downstream effects before replay. Restoring code, files, and database together is different from replaying one event or reconciling one external effect.

Authenticate delivery, then authorize the business action

A valid provider signature proves only the configured delivery path under that provider contract. It does not prove that the event may read or change any tenant, account, order, payment, or private record.

• Verify the exact signed bytes before parsing, apply the provider timestamp and replay rules before deduplication, and use the provider verifier when one is available.
• Keep signing secrets, public-key configuration, outbound tokens, and rotation material server-side with separate scopes and owners. Never place secrets in callback URLs or logs.
• Bound request size, allowed content type, event types, field lengths, nesting, and batch count. Validate every required record identity before the first durable business write.
• Store provider event or delivery identity under a durable unique constraint. Keep a safe payload fingerprint so changed content cannot hide behind an accepted identity.
• Apply tenant, actor, resource, state-transition, and expected-version authorization after transport authentication and before the business change.
• Minimize retained payloads, redact routine logs, define retention and deletion ownership, and treat raw bodies as potentially sensitive even when the signature is valid.
• Rate-limit resource-intensive validation and operator replay paths without exposing thresholds that materially aid abuse. Never let a public request choose an arbitrary replay identity.
• Make manual replay a reviewed operation with reason, actor, target identity, expected current state, effect reconciliation, outcome, and audit record.

Webhook receiver questions