Idempotency Keys and Exactly-Once Delivery

A customer taps Pay. The payment is captured, but the connection disappears before the response reaches the phone. Did payment fail, or only the response? Retrying may collect twice; refusing may leave a paid order displayed as unpaid.

This uncertainty is not an edge case. Networks can lose responses, processes can crash after committing, brokers can redeliver, and consumers can finish work before acknowledging it. No timeout can tell the caller which side of a commit boundary the operation reached.

Idempotency keys are a practical answer: the caller gives a logical operation a stable identity, and the server remembers its outcome. They do not make the network exactly-once. They make repeated attempts converge on one business result within a deliberately defined scope. That distinction is the foundation of a reliable design.

Delivery guarantees are claims about boundaries #

Delivery terminology is useful only when it names both the mechanism and the boundary being protected. A broker may deliver a record once while an application performs an external side effect twice. An HTTP gateway may retry once while a queue retries five more times downstream. “Exactly once” without a boundary is marketing, not an invariant.

At-most-once fits when dropping work is cheaper than repeating it, such as telemetry. At-least-once fits orders and durable events because retrying is safer than silent loss. It shifts duplicate handling where business identity is available.

Some systems legitimately provide exactly-once processing. Kafka transactions can atomically consume and publish Kafka records under specific configuration. A relational database can atomically update domain and deduplication rows. Neither guarantee automatically includes an email provider, second database, or card network. Crossing that boundary requires another idempotency contract or compensation.

The useful invariant is narrower and testable: for a given operation identity and payload, this service commits at most one result and returns the same completed result to every retry.

An idempotency key is a business operation identity #

An idempotency key should identify the caller’s intent, not an individual network attempt. A mobile client creates a random key when the user starts checkout, persists it with the pending action, and reuses it after timeouts, reconnects, and app restarts. Generating a new key for each retry defeats the mechanism.

The server must place the key in a scope. A globally unique key is simple but often unnecessary. A practical uniqueness constraint might be (tenant_id, route, idempotency_key) or (account_id, operation_type, key). Scoping prevents one tenant from colliding with another and prevents a key used for POST /payments from accidentally replaying a response from POST /refunds.

The key alone is not enough. Store a cryptographic hash of the normalized request payload. If a caller reuses a key with a different amount, currency, or destination, reject it with a conflict instead of returning an unrelated old response. The hash binds identity to intent.

Normalization must be deterministic. Raw JSON is fragile because whitespace and key order can differ without changing meaning. Validate first, create a canonical representation with sorted keys and explicit defaults, then hash it. Exclude transport fields such as trace IDs, but include every field that changes the business effect. Define handling for numbers, Unicode, absent versus null, and ordered arrays.

A useful record contains:

Do not cache every failure blindly. A stable business rejection such as “insufficient funds” may be a completed result if the contract says the same operation cannot be changed. A transient database outage should normally leave no completed record, so the same key can retry. Validation failures that occur before claiming the key need not be stored at all.

Model the record as a state machine #

The safest design for a local database effect claims the idempotency key, changes domain state, and stores the replayable response in the same database transaction. A unique constraint on (scope, key) is mandatory. Application-level “check then insert” is racy: two workers can both observe absence and both perform the effect.

The following TypeScript keeps database details behind a transaction interface, but its claim operation has a strict implementation requirement: use an atomic insert-on-conflict plus row lock, or an equivalent serializable operation. It must not be two unrelated queries.

import { canonicalJson, sha256 } from './hash.js';
import type { Database, StoredResponse, Transaction } from './database.js';

type IdempotencyState = 'processing' | 'succeeded';

type IdempotencyRecord = Readonly<{
  scope: string;
  key: string;
  requestHash: string;
  state: IdempotencyState;
  response?: StoredResponse;
  resourceId?: string;
  leaseExpiresAt: Date;
  expiresAt: Date;
}>;

type ClaimResult =
  | { kind: 'claimed'; record: IdempotencyRecord }
  | { kind: 'replay'; response: StoredResponse }
  | { kind: 'busy'; retryAfterSeconds: number }
  | { kind: 'mismatch' };

type CreateOrderInput = Readonly<{
  customerId: string;
  currency: 'USD' | 'EUR' | 'VND';
  totalMinor: number;
  items: ReadonlyArray<{ sku: string; quantity: number }>;
}>;

export async function createOrderOnce(
  database: Database,
  tenantId: string,
  idempotencyKey: string,
  input: CreateOrderInput,
): Promise<StoredResponse> {
  const scope = `${tenantId}:orders:create`;
  const requestHash = sha256(canonicalJson(input));

  return database.transaction('serializable', async (transaction) => {
    const now = await transaction.clock.now();
    const claim = await transaction.idempotency.claim({
      scope,
      key: idempotencyKey,
      requestHash,
      leaseExpiresAt: new Date(now.getTime() + 30_000),
      expiresAt: new Date(now.getTime() + 24 * 60 * 60 * 1000),
    });

    switch (claim.kind) {
      case 'mismatch':
        return {
          status: 409,
          body: { code: 'IDEMPOTENCY_KEY_REUSED' },
        };
      case 'busy':
        return {
          status: 409,
          headers: { 'retry-after': String(claim.retryAfterSeconds) },
          body: { code: 'OPERATION_IN_PROGRESS' },
        };
      case 'replay':
        return claim.response;
      case 'claimed':
        return commitOrder(transaction, claim.record, tenantId, input);
    }
  });
}

async function commitOrder(
  transaction: Transaction,
  record: IdempotencyRecord,
  tenantId: string,
  input: CreateOrderInput,
): Promise<StoredResponse> {
  const order = await transaction.orders.insert({
    tenantId,
    customerId: input.customerId,
    currency: input.currency,
    totalMinor: input.totalMinor,
    items: input.items,
  });

  const response: StoredResponse = {
    status: 201,
    headers: { location: `/orders/${order.id}` },
    body: { orderId: order.id, status: order.status },
  };

  await transaction.idempotency.succeed({
    scope: record.scope,
    key: record.key,
    requestHash: record.requestHash,
    resourceId: order.id,
    response,
  });

  return response;
}

If the process crashes before commit, the order and idempotency claim both roll back. If it crashes after commit but before sending HTTP bytes, the retry finds succeeded and replays the stored response. There is no committed processing row in this single-transaction variant; the state still matters when work is deliberately asynchronous or spans multiple short transactions.

Do not hold a database transaction open while calling a remote payment API. Locks held across network latency reduce throughput and still cannot atomically commit the remote effect. Instead, commit a durable intent, let a worker call the provider using the same operation key, and record the result. The provider must honor that key, or the integration must reconcile ambiguous outcomes before retrying.

Concurrent duplicates must have one winner #

Two identical requests may arrive simultaneously. The database must arbitrate them.

The losing request can wait briefly for the winner, return 409 Operation In Progress with Retry-After, or return 202 Accepted with a status URL. Choose one contract and document it. Waiting is convenient for fast operations; 202 is better for long-running jobs. Returning an ordinary 500 encourages aggressive retries and hides the fact that useful work is already running.

A lease is needed when processing is committed before the effect finishes. It prevents permanent blockage after a worker dies. A contender may take over only after lease_expires_at, ideally with a fencing token or monotonically increasing attempt number. The worker must verify that token before writing completion so a slow, expired worker cannot overwrite the newer owner.

Even then, lease expiry does not prove the old worker performed no external effect. Suppose it charged a card and crashed before recording success. The takeover worker must query the provider by operation key or reconciliation reference before charging again. Time is not evidence of non-execution.

Scope, retention, and TTL are correctness decisions #

Deduplication lasts only as long as the record. A 24-hour TTL means “effectively once for retries that arrive within 24 hours,” not forever. The TTL must exceed the longest realistic retry horizon: client offline duration, broker retention, manual replay windows, and delayed job retries. Financial operations may need records retained for months or moved to an audit store; high-volume telemetry may need minutes.

Deleting records creates a reuse risk. After expiration, the same key looks new and can recreate the effect. Reduce that risk with high-entropy keys, a server-defined scope, documented retention, and a business uniqueness constraint where one exists. An order table might also enforce a unique (merchant_id, client_order_id), providing durable protection independent of the idempotency cache.

TTL should be based on server time. Do not trust a client timestamp to decide whether a key is stale. Cleanup should operate in bounded batches using an index on expires_at, avoid large table locks, and expose lag metrics. Partitioning by creation date can make high-volume deletion cheaper, but only if the retention policy aligns with partition boundaries.

Store the minimum response necessary to reproduce the API contract. Full response bodies are convenient but may contain personal data, secrets, or values whose retention is regulated. Often a status code, selected headers, resource ID, and versioned response snapshot are sufficient. Encrypt sensitive records, apply access controls, and include deletion requirements in the data lifecycle design.

Watch cardinality and hot spots. Keys scoped to one popular tenant can concentrate writes in an index. Random key material generally distributes well, while sequential keys may not. Capacity planning must include records created by rejected or abandoned workflows, not only successful business objects.

Outbox and inbox connect local truth to messages #

An idempotent HTTP endpoint solves only its local operation. Publishing an event introduces the dual-write problem: committing an order and publishing OrderCreated are two independent actions. Publishing first can announce an order that later rolls back. Committing first can leave an order with no event if the process crashes.

The transactional outbox writes the domain change and an outbox row in the same database transaction. A relay later publishes the row at least once. It marks the row sent after broker acknowledgement, so a crash between publish and mark may publish it again. That duplication is intentional and must be handled downstream.

import type { OutboxStore, Publisher } from './messaging.js';

export async function relayOutbox(
  outbox: OutboxStore,
  publisher: Publisher,
): Promise<void> {
  const messages = await outbox.claimBatch({ limit: 100, leaseMs: 30_000 });

  for (const message of messages) {
    try {
      await publisher.publish(message.topic, message.payload, {
        messageId: message.id,
      });
      await outbox.markPublished(message.id, message.leaseToken);
    } catch (error) {
      await outbox.releaseForRetry(message.id, message.leaseToken, error);
    }
  }
}

At the consumer, an inbox table applies the symmetric rule. In one transaction, insert the message ID into an inbox table with a unique (consumer, message_id) constraint, update local domain state, and commit. If the insert conflicts, acknowledge the duplicate without applying the state transition again. The deduplication identity should normally be the producer’s stable event ID, not a broker delivery tag that changes on redelivery.

This composition gives an honest end-to-end story: the producer commits each local intent once, the outbox publishes at least once, the broker delivers at least once, and each consumer commits each event to its local database once. External notifications still need provider idempotency or a business rule. Email may be acceptable at-least-once with duplicate suppression; money movement demands stronger reconciliation.

Operate the ambiguity, not just the happy path #

Reliability depends on seeing records that stop moving. Track claim rate, replay rate, payload mismatches, in-progress age, lease takeovers, outbox backlog, publish attempts, inbox duplicate rate, and cleanup lag. Alert on the oldest unprocessed record rather than backlog count alone: a small poisoned partition can be more serious than a large but draining burst.

Classify failures explicitly. Validation errors are terminal before execution. Business rejections may be stable completed outcomes. Timeouts and dependency failures are retryable but need exponential backoff and jitter. Poison messages need a dead-letter or quarantine path with enough context for diagnosis. Manual replay tools must preserve the original message ID; assigning a new ID bypasses the inbox by definition.

Operational repair must also respect ownership. An administrator should not simply change processing to succeeded. Recovery should inspect the domain resource and any remote provider using correlation identifiers, then either attach the known result, release the claim, or compensate. Every repair action should be audited.

For schema changes, deploy readers that understand both record formats before new writers. Version stored responses as APIs evolve. If old responses cannot be preserved safely, document a shorter replay window and return a resource reference afterward.

Finally, rehearse the dangerous timing windows: crash before local commit, crash after commit before response, duplicate requests in parallel, outbox publish before acknowledgement, consumer commit before acknowledgement, lease expiry during a slow provider call, and cleanup racing with a late retry. These tests reveal guarantees more accurately than a diagram labeled “exactly once.”

Takeaways #

Exactly-once delivery is rarely an end-to-end property of a distributed system. It is usually a bounded atomicity guarantee plus application-level deduplication at every boundary. State that boundary precisely.

Use a stable operation key, scope it by tenant and operation, bind it to a canonical payload hash, and let a database uniqueness constraint choose the concurrent winner. Commit the domain effect and replayable result together whenever they share a database. For asynchronous work, use explicit states, leases, fencing, and reconciliation rather than assuming an expired timer means nothing happened.

Connect database state to messaging with an outbox, protect consumers with an inbox, and expect both relays and brokers to redeliver. Choose TTL from the real retry horizon, clean up incrementally, and monitor stuck age as well as volume.

The goal is not to prevent duplicate packets. It is to make duplicate attempts observable and unable to create a second business outcome.