Idempotent APIs and webhooks: making retries safe

Every network call can fail in a way that leaves the caller unsure whether it succeeded. A timeout, a dropped connection, a 502 from a proxy in front of a request that actually completed. The client has one reasonable response: try again.

Which means every write endpoint will eventually receive the same request twice, whether or not it was designed for it. The question is only whether the second one is harmless or creates a duplicate charge, a duplicate lead, a duplicate order.

Retries are not optional, so duplicates are not either

The instinct is to make retries rarer — longer timeouts, better error handling, fewer moving parts. That treats the symptom. Retries come from clients you do not control: mobile apps on bad connections, third-party webhook senders, job queues, load balancers, users double-clicking a button.

The useful framing: make a repeated request cost nothing, then retry freely. Idempotency first, retry logic second. In that order, aggressive retries are a reliability feature. In the other order, they are a data-corruption feature.

Case 1: the request has a natural key

Most of the time, something in the payload already identifies the operation uniquely — a provider's record id, an order number, a transaction reference. That is a natural key, and the database can enforce it:

CREATE TABLE lead (
  id           BIGINT UNSIGNED PRIMARY KEY AUTO_INCREMENT,
  source       VARCHAR(32)  NOT NULL,
  source_ref   VARCHAR(64)  NOT NULL,   -- the sender's own id
  payload      JSON         NOT NULL,
  status       VARCHAR(24)  NOT NULL DEFAULT 'new',
  created_at   DATETIME     NOT NULL,
  UNIQUE KEY uniq_source_ref (source, source_ref)
);

Then write through an upsert rather than an insert:

public function store(string $source, array $record): void
{
    Yii::$app->db->createCommand()->upsert('lead', [
        'source'     => $source,
        'source_ref' => $record['id'],
        'payload'    => json_encode($record, JSON_UNESCAPED_UNICODE),
        'created_at' => $record['created_at'],
    ], [
        // Refresh only what the sender owns. Never reset status — a human may have worked it.
        'payload' => new Expression('VALUES(payload)'),
    ])->execute();
}

Two details matter more than they look.

Enforce it in the database. The application-level version — select, check, insert — has a race window between the check and the insert. Two concurrent deliveries will land in it eventually, and under load, often. A unique constraint is atomic; no interleaving defeats it.

Be careful what the update clause touches. The second delivery may arrive after a human has already acted on the record. An upsert that overwrites status drags closed work back into the queue — the same bug the idempotency was meant to prevent, wearing a different costume.

Case 2: there is no natural key

Some operations genuinely have nothing unique in them. "Charge this card ₹2,000" is the same request whether it is a retry or a second, intentional purchase. Only the client knows which.

So the client supplies the distinction — an idempotency key, generated once per logical operation and resent on every retry:

POST /api/payments
Idempotency-Key: 6a1f0c7e-2b43-4f9a-9d1c-0e2f5a7b8c90

The server records the key with the outcome:

CREATE TABLE idempotency (
  id            VARCHAR(64) PRIMARY KEY,     -- the key
  request_hash  CHAR(64)    NOT NULL,        -- fingerprint of the payload
  status_code   SMALLINT    NULL,
  response      JSON        NULL,
  state         ENUM('in_progress','done') NOT NULL,
  created_at    DATETIME    NOT NULL
);

The handler then becomes:

public function handle(string $key, array $payload): Response
{
    $hash = hash('sha256', json_encode($payload));

    // Claim the key. A duplicate request loses this race by design.
    $claimed = $this->db->createCommand(
        'INSERT IGNORE INTO idempotency (id, request_hash, state, created_at)
         VALUES (:id, :hash, "in_progress", NOW())'
    )->bindValues([':id' => $key, ':hash' => $hash])->execute();

    if (!$claimed) {
        $row = $this->findKey($key);
        if ($row['request_hash'] !== $hash) {
            // Same key, different body — a client bug worth surfacing loudly.
            return new Response(422, ['error' => 'idempotency key reused with a different payload']);
        }
        if ($row['state'] === 'in_progress') {
            return new Response(409, ['error' => 'request already in progress']);
        }
        return new Response($row['status_code'], json_decode($row['response'], true));
    }

    $result = $this->performWork($payload);
    $this->completeKey($key, $result);
    return $result;
}

Three behaviours fall out of that, and all three matter:

• A retry after success replays the stored response. The work happens once; the client still gets its answer.
• A retry while the first is still running gets a 409 rather than racing it. Without the in_progress state, two concurrent retries both proceed.
• A key reused with a different payload is rejected. That is a client bug, and silently returning the old response hides it.

Webhooks: you are the one retried

This is the foundation a lead pipeline rests on — the reason custom CRM development treats it as a day-one concern rather than hardening. Everything above assumes you are the server. When you consume webhooks, you are on the receiving end of someone else's retry policy — and senders retry generously, including after your endpoint returned 200 but the connection dropped before they saw it.

Three rules make consumption safe:

Store the sender's id with a unique constraint. This is case 1. A replayed delivery converges on one row.

Acknowledge fast, process after. Write the raw payload, return 200, and do the real work in a queue. A handler that calls three services before responding will eventually exceed the sender's timeout and be retried while it is still working.

Verify the signature before trusting any of it. An endpoint that accepts unsigned payloads is an endpoint anyone can write to.

// Constant-time comparison: a plain === leaks timing information.
if (!hash_equals($expected, $provided)) {
    return new Response(401);
}

And assume delivery will fail sometimes regardless. A webhook is a delivery attempt, not a guarantee — if the records matter, pair it with a scheduled reconcile that pulls from the sender's API and upserts through the same path. Because the write is idempotent, the reconciler costs nothing when the webhook already worked.

What to watch

Not errors — a correctly idempotent system produces very few. Watch the duplicate rate: how often a request arrives whose key or natural key already exists.

A steady background level is healthy; it means retries are happening and being absorbed. A sudden spike means something upstream is failing and retrying hard, which is exactly the moment you want to know, and exactly the moment nothing is throwing an exception.

The takeaway

Idempotency is not a hardening task for later. It is the property that makes every other reliability measure safe to add, and retries, queues and reconcilers are all dangerous without it.

The implementation is almost always one of two things: a unique constraint on a key the payload already contains, or a key the client supplies with the stored response behind it. Both are a day of work. Both are far cheaper than the afternoon spent deduplicating a production table by hand.