Designing Reliable Webhooks

A webhook is often described as an HTTP request sent when something happens. That description captures the transport and misses the engineering problem. The provider and consumer are independent systems with different deployments, clocks, failure modes, and operators. Networks lose responses, receivers become unavailable, secrets rotate, events arrive twice, and a customer eventually asks why one particular notification did not produce the expected action.

Reliable webhooks therefore need a protocol, not just a POST endpoint. The protocol must define what an event means, how its bytes are authenticated, when delivery is considered complete, how failures are retried, what ordering is promised, and which evidence both sides can use to debug a disputed delivery. Idempotent consumption is one part of that contract, but it cannot compensate for an ambiguous event, an unverifiable signature, or a provider that discards delivery history.

This article designs that protocol around an order platform sending order.fulfilled events. The goal is not impossible exactly-once delivery. It is an honest at-least-once service whose duplicates, delays, key changes, and operator actions have defined outcomes.

Define the Delivery Contract Before the Endpoint #

Begin with externally visible guarantees. A useful provider contract answers five questions:

1. Which state change creates each event, and can that decision later be reversed?
2. Is delivery at most once, at least once, or best effort?
3. What response status and deadline count as acknowledgement?
4. How long will automatic retries and manual redelivery remain available?
5. Is ordering guaranteed, scoped to one resource, or explicitly unspecified?

For the example, order.fulfilled means the order first entered the fulfilled state. The event is immutable after creation. Delivery is at least once: the provider may send the same event multiple times until one attempt receives an accepted response. Any 2xx response received within ten seconds acknowledges that endpoint attempt; redirects are not followed. The provider retries for a documented retention window and makes the event available for operator-triggered redelivery during that window. Events for one order carry a monotonic resource version, but transport order is not guaranteed.

Those choices should appear in documentation and tests. Consumers otherwise infer accidental behavior. If a provider currently sends events in order because it has one worker, customers may build on that observation. Adding workers later becomes a breaking change even though no schema changed.

Separate event creation from endpoint delivery. One business occurrence produces one event ID. Sending it to three subscribed endpoints creates three delivery streams, each with independent attempts and status. Disabling one endpoint must not alter the identity or payload seen by the others. This distinction also makes records intelligible: an event answers “what happened?” while a delivery answers “what happened when we tried to tell this endpoint?”

Version the contract deliberately. A schema version belongs in the envelope, while event-type evolution should preserve old field meanings. Additive fields are usually compatible only if consumers are instructed to ignore unknown fields. Removing a field, changing units, or turning an optional field into a required assumption needs a new version or event type. The endpoint URL is not a versioning strategy.

Design an Envelope with Stable Meaning #

Every event needs an envelope that is useful before the consumer understands its domain payload. A compact example is:

{
  "id": "evt_01JX8M4Z6F7Q2P3K8N5T",
  "type": "order.fulfilled",
  "schemaVersion": 1,
  "occurredAt": "2026-07-15T14:23:08.441Z",
  "resource": {
    "kind": "order",
    "id": "ord_7421",
    "version": 8
  },
  "data": {
    "orderId": "ord_7421",
    "warehouseId": "wh_03",
    "fulfilledAt": "2026-07-15T14:23:07.912Z"
  }
}

id identifies the immutable event across all attempts. It is the receiver’s deduplication key and the operator’s primary lookup handle. type selects semantics, not merely a serializer. occurredAt records when the source fact became true; it must not be replaced on retries. A separate attempt timestamp belongs in delivery metadata or signature headers. schemaVersion tells a consumer how to validate data.

The resource identifier and version make stale-event handling possible without pretending the network preserves order. A consumer maintaining a projection can reject version 7 after it has committed version 8, or fetch current state when it detects a gap. This is stronger than comparing timestamps, which can tie, arrive with differing precision, or describe business time rather than state order.

Choose payload style by semantics. A thin event containing only orderId keeps the webhook small and encourages the receiver to fetch current state, but it creates an API dependency and cannot reconstruct historical state later. A snapshot event is self-contained but can expose unnecessary data and becomes stale after creation. A delta is compact but requires strict ordering and complete history. Many systems use a bounded snapshot containing the fields relevant to that event, plus an authenticated API for consumers that need fresh detail.

Do not put credentials, internal error messages, or unrelated personal data in the payload. Webhook bodies are copied into queues, logs, retry stores, and customer debugging tools. Data minimization reduces every one of those exposure surfaces. Publish JSON field types, nullability, size limits, timestamp format, and canonical examples. A sample payload without a schema is an illustration, not a contract.

Sign the Exact Bytes and Defend Against Replay #

TLS authenticates the server connection during transit, but it does not let a receiver distinguish the provider from another caller that can reach the endpoint. Sign every request with a per-endpoint secret. A common construction signs a version marker, creation timestamp, and exact body bytes:

Send the timestamp and one or more signatures in a structured header, for example:

Webhook-Signature: t=1784125388,v1=4c18...a09f,v1=7bd2...319a
Webhook-Id: evt_01JX8M4Z6F7Q2P3K8N5T

The receiver must verify against the raw request bytes before JSON parsing. Parsing and reserializing can alter whitespace, key order, escaping, or numeric representation and invalidate a legitimate signature. Read the body through a size-limited raw-body path, calculate the HMAC, decode both values to equal-length byte arrays, and use a constant-time comparison. Reject unsupported signature versions rather than silently guessing an algorithm.

The signed timestamp bounds replay. The receiver checks that it falls within an allowed tolerance relative to its clock, then verifies the signature. This only limits how long a captured request can be freshly presented; it does not replace event-ID deduplication. A legitimate provider retry outside the tolerance carries a new attempt timestamp and signature over the same event body. Clock monitoring matters because a badly skewed receiver can reject every valid delivery or accept old attempts for too long.

Secret rotation requires overlap. During rotation, the provider signs with the new secret and may include a signature from the retiring secret for a bounded period. The receiver accepts either configured key, records which key ID verified, and removes the old one after deployment has propagated. Never reveal the secret again in delivery logs. Store it encrypted, redact headers, and make rotation an audited endpoint operation.

Replay protection is not endpoint verification. When registering a URL, prove that its operator controls it. One approach sends a short-lived random challenge that the endpoint must return, authenticated through the same secret or a setup token. Validate schemes, block embedded credentials, resolve and restrict destinations according to the product’s server-side request forgery policy, and re-evaluate redirects rather than following them automatically. Registration is a security boundary because the provider’s delivery workers can otherwise become network probes.

Model Delivery as Durable State #

An event should be committed with the source change, or derived from a durable source log, before any network request is attempted. Writing the order and then making an inline callback creates a gap: a process crash can commit fulfillment without ever creating its notification. An outbox table solves that local atomicity problem by storing the event in the same database transaction as the order transition. A dispatcher later creates endpoint delivery records.

Persist an attempt before or while leasing it, including event ID, endpoint ID, attempt number, selected URL revision, request timestamp, body digest, status or network error class, response duration, a bounded response excerpt, and the computed next action. A worker crash after sending but before recording the response leaves the outcome unknown. The correct recovery is another attempt, not an assertion that the first failed. This uncertainty is why consumers must tolerate duplicates.

Use leases so abandoned in-flight work becomes eligible again. A lease is worker coordination, not a claim that the HTTP request stopped when the lease expired. Keep concurrency bounded per endpoint and globally. One failing customer must not occupy every socket or retry slot. Endpoint state can transition to paused after sustained failures, but pausing should be visible and reversible rather than silently dropping events.

A 2xx acknowledges receipt under the provider contract. The consumer should return it only after durably recording the event, commonly by inserting an inbox row and queueing local work in one transaction. Waiting for a slow email, report, or third-party API before responding couples the provider’s retry behavior to the consumer’s entire workflow. Conversely, returning 200 before durable acceptance can lose the event if the consumer crashes immediately afterward.

Retry by Failure Class, Not by Hope #

Retries should recover transient conditions without amplifying permanent ones. Classify outcomes explicitly:

Providers may choose to retry selected 4xx responses briefly during deployments, but that is a documented policy, not a universal truth. Honor a valid Retry-After within service limits for 429 or 503. Otherwise use exponential backoff with jitter and a maximum interval. Jitter prevents thousands of endpoints from being retried on the same boundary after a regional outage.

Every attempt uses the same event ID and body but a fresh attempt timestamp and signature. Mutating the payload during retries destroys the stable unit that consumers deduplicate and operators compare. If a correction is necessary, emit a new event with its own identity and semantics.

Set a finite attempt count or delivery horizon. Infinite retries retain data forever, consume capacity, and obscure endpoints that require human repair. On exhaustion, mark the delivery failed, expose the reason, and notify the endpoint owner through a channel independent of that webhook. Manual redelivery should create a new attempt under the same event and record the actor and reason. It should not edit old attempt history.

Retry scheduling must also respect event age and endpoint backlogs. If an endpoint recovers after a day, sending ten thousand old events at full speed can take it down again. Drain with a per-endpoint rate limit, preserve fair access for current events, and let consumers fetch or replay a bounded event range when bulk recovery is more efficient.

Handle Duplicates and Ordering at the Receiver #

The receiver’s critical section is “record once, acknowledge after record.” For a relational store, an inbox table with a unique provider-and-event key makes that decision atomic:

BEGIN;

INSERT INTO webhook_inbox (
  provider, event_id, event_type, resource_id, resource_version, payload
) VALUES (
  'orders', :event_id, :event_type, :resource_id, :resource_version, :payload
)
ON CONFLICT (provider, event_id) DO NOTHING;

-- Enqueue local processing only when the insert created a row.
INSERT INTO local_jobs (kind, source_event_id)
SELECT 'apply_order_event', :event_id
WHERE EXISTS (
  SELECT 1 FROM webhook_inbox
  WHERE provider = 'orders' AND event_id = :event_id AND processed_at IS NULL
);

COMMIT;

Production code must ensure the job insert is itself unique on source_event_id; the abbreviated query alone can enqueue twice under concurrent requests. The important webhook rule is that deduplication keys come from the provider’s stable event identity and are scoped by provider or account. A hash of the body is weaker: two legitimate events can share a payload, and inconsequential serialization changes can produce different hashes.

Deduplication prevents repeating acceptance work, but domain effects still need explicit invariants. Updating a projection to resource version 8 can be conditional on its current version being below 8. Sending a customer message may need a unique business key such as (notification_kind, order_id, fulfilled_version). This is where webhook processing meets idempotency, but the webhook contract supplies the identity and ordering evidence needed to apply it.

Do not infer global ordering from arrival order. Parallel dispatch, retries, and network delays all reorder requests. If the provider promises per-resource versions, buffer a small gap or fetch current state when version 10 arrives before 9. If it promises no sequence at all, design events as independently applicable facts or use the provider API as the authoritative current-state read. Waiting forever for a missing sequence number turns one lost event into permanent blockage.

Give Operators a Shared Debugging Story #

Webhook support becomes expensive when the provider can only say “we sent it” and the consumer can only say “we did not see it.” Build a delivery explorer keyed by event ID. Show the event type, creation time, endpoint and URL revision, attempt timeline, request body as permitted by data policy, body digest, signature key ID, status or normalized network error, duration, next retry, and final state. Redact secrets and truncate response bodies aggressively.

Include an attempt identifier in a request header and return it in logs or support exports. The event ID correlates semantic duplicates; the attempt ID distinguishes individual HTTP exchanges. Trace IDs may help internally, but they should not replace stable public identifiers because tracing samples can be dropped.

Expose endpoint health as actionable state: recent success rate, oldest pending event, queue depth, last accepted attempt, last terminal error, current pause reason, and secret-rotation status. Avoid ranking customers by a single average latency. A mostly fast endpoint with recurring timeouts needs different action from one consistently responding in eleven seconds against a ten-second deadline.

Operational controls need authorization and audit history. Useful controls include pause, resume, rotate secret, test endpoint, redeliver one event, and request replay over a bounded interval. A synthetic test event must be clearly marked and must not reuse a real event ID. Replays should preserve original event identities so receiver deduplication remains effective, while recording the replay request separately.

Retention is part of the contract. State how long payloads, attempt metadata, and redelivery controls remain available. Sensitive payloads may need shorter retention than status metadata. If an event is no longer replayable, the UI and API should say so instead of presenting a button that reconstructs a subtly different payload from current state.

Verify the Protocol End to End #

Test provider and consumer behavior at protocol boundaries, not only happy-path handlers. Publish signature fixtures containing the raw bytes, timestamp, secret, and expected header so consumers can verify implementations in any language. Include whitespace, Unicode, empty objects, multiple signatures, expired timestamps, malformed encodings, and a body changed by one byte. Confirm that parsing occurs only after size and signature checks.

Provider integration tests should simulate connection refusal, DNS errors, response timeout, truncated responses, every relevant status class, Retry-After, worker termination after send, lease expiry, and secret rotation. Use a controlled clock to assert retry windows and jitter bounds without sleeping. Assert that all attempts retain one event ID and body digest, terminal outcomes stop automatic scheduling, and manual redelivery leaves immutable history.

Receiver tests should deliver the same event concurrently, deliver version 8 before version 7, retry after the receiver commits but before it returns a response, and replay an event after local processing succeeds. Verify one durable inbox record and one domain effect. Test recovery from a processing failure separately from HTTP acceptance; once the event is durably accepted, the consumer’s own job system owns retries.

Before releasing a schema change, run consumer-driven contract fixtures for supported versions and compare serialized payloads. During operations, alert on sustained delivery failure, growing oldest-pending age, retry amplification, signature rejection spikes, and exhausted endpoints. Rehearse secret rotation and outage recovery with a non-production endpoint, including backlog draining.

The durable design is straightforward: create immutable events with stable identities, sign exact bytes, durably track each endpoint attempt, retry only classified transient outcomes, and make duplicates and reordering explicit. Then give both operators the same identifiers and evidence. That protocol cannot make two independent systems fail as one, but it can make every ambiguous boundary visible, bounded, and recoverable.