Your Retry Worked. That Is How You Got Two Rows.

A retry looks harmless because the code usually changes by one line: catch the timeout, wait, try again.

That line can also create two invoices, two customer records, two feature rows, or two downstream notifications.

The uncomfortable case is not when a request clearly fails. It is when the caller times out after the server has already completed the work but before the response arrives. From the caller's point of view, nothing happened. From the system's point of view, the mutation is already committed. A blind retry turns one business event into two database effects.

That is the problem idempotency solves.

Idempotency is a business guarantee, not a retry setting

An operation is idempotent when repeating the same intended request produces no additional side effect. The first attempt may create a row. The fifth attempt should not create rows two through five.

The important phrase is same intended request. Two requests with identical payloads are not necessarily duplicates. A customer may legitimately buy the same item twice. Two data files can contain the same values but represent different deliveries. A hash of the payload therefore cannot always tell you whether the caller meant “retry the first operation” or “perform a new operation.”

AWS's guidance on making retries safe with idempotent APIs recommends a caller-provided request identifier precisely because it carries intent. The client creates the identifier before the first attempt and reuses it for every retry of that operation.

The invariant is simple:

One business operation gets one stable key. Every retry carries that same key.

A new business operation gets a new key, even if every other field is identical.

The failure window that creates duplicates

Imagine an ingestion service receiving event EVT-8421:

1. The service writes the event to the warehouse.
2. The database commits successfully.
3. The network connection drops before the service returns 200 OK.
4. The producer sees a timeout and retries.
5. The service inserts the same event again.

Nothing in this sequence is exotic. Every component did something reasonable. The duplicate appears because the system has no durable way to connect attempt two with attempt one.

“At least once” delivery makes this explicit: a message should not be lost, but it may be delivered again. That is often the correct transport guarantee. The destination still needs a rule for what a repeated event means.

Pattern 1: Generate the key before the first attempt

The best key usually comes from the business event, not from the retrying worker.

Good candidates include:

• a payment attempt ID generated by the client;
• a source-system event ID;
• a file delivery ID plus row number;
• a job run ID plus logical partition;
• an order ID plus operation type;
• a deterministic key built from stable business identifiers when the domain truly guarantees uniqueness.

A random UUID created inside every retry is useless. Each attempt will look new. The key must survive process restarts, queue redelivery, and worker reassignment.

Stripe's idempotent request documentation illustrates the contract clearly: the client supplies a unique key, retries reuse it, and a reused key with different parameters is rejected. That final rule matters. If the payload changes while the key stays the same, the safest response is usually an error—not a guess about the caller's intent.

Pattern 2: Enforce uniqueness where the write happens

Checking for a duplicate in application code and then inserting is not enough:

-- Race condition: two workers can both see no row.
SELECT 1 FROM processed_events WHERE event_id = 'EVT-8421';
INSERT INTO processed_events (...);

Between the check and the insert, another worker can do the same thing. This is a classic time-of-check/time-of-use race.

Put the invariant in the database:

CREATE TABLE processed_events (
    event_id      text PRIMARY KEY,
    payload_hash  text NOT NULL,
    processed_at  timestamptz NOT NULL DEFAULT now()
);
 
INSERT INTO processed_events (event_id, payload_hash)
VALUES ('EVT-8421', 'sha256:...')
ON CONFLICT (event_id) DO NOTHING;

PostgreSQL documents that INSERT ... ON CONFLICT can choose an alternative to a uniqueness error, and that ON CONFLICT DO UPDATE provides an atomic insert-or-update outcome under concurrency. The unique constraint is doing the hard work: two workers may race, but only one can establish the key.

Be precise about what DO NOTHING means. It prevents the duplicate row. It does not automatically tell the caller whether the existing row contains the same intended operation. Store a payload hash or the normalized request parameters and compare them. Same key, different intent should fail loudly.

Pattern 3: Make deduplication and the side effect atomic

A dedupe table can still lie if it is committed separately from the work.

Failure mode A:

1. Record the idempotency key.
2. Crash before writing the business row.
3. The retry sees the key and skips work that never happened.

Failure mode B:

1. Write the business row.
2. Crash before recording the key.
3. The retry repeats the business write.

When both records live in the same database, write them in one transaction:

BEGIN;
 
INSERT INTO processed_events (event_id, payload_hash)
VALUES ('EVT-8421', 'sha256:...')
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
 
-- Continue only if the insert returned a row.
INSERT INTO fact_claims (...);
 
COMMIT;

The real implementation must condition the business write on whether the key was newly inserted. The point is that the key and the effect commit together.

External side effects—sending email, charging a card, calling another API—cannot usually share that database transaction. That is where an outbox pattern helps: commit the business change and an outbox message atomically, then let a relay deliver the message with its own stable key. The downstream service must also be idempotent. Reliability is an end-to-end property; one idempotent component cannot make an entire chain safe.

Pattern 4: Define the replay window

Idempotency records cannot always live forever. Stripe says keys can be pruned after at least 24 hours. Your appropriate window depends on the longest credible retry, redelivery, backfill, and disaster-recovery interval.

A ten-minute window may cover HTTP retries but fail during a next-day queue replay. A seven-day window may cover ordinary incidents but fail when a month-old partition is backfilled.

Document four things:

• how long a key remains authoritative;
• what response a duplicate receives;
• what happens when the same key carries different parameters;
• what happens after the key expires.

Without that contract, “idempotent” is only a local implementation detail.

Pattern 5: Treat exactly-once claims as boundary-specific

Kafka's current design documentation distinguishes at-most-once, at-least-once, and exactly-once processing. Kafka can provide exactly-once behavior when reading, processing, and writing within Kafka using transactions and read-committed consumers. Its documentation also notes that writing to an external destination generally requires cooperation from that destination. See Kafka's message delivery semantics.

That boundary is easy to miss. A transactional producer can prevent duplicate records in an output topic, but it cannot magically make a separate REST API or warehouse write atomic with the Kafka offset. For an external relational database, one robust pattern is to store the consumer's progress and the output in the same database transaction. Another is to make the destination write idempotent using the event ID.

“Exactly once” is not a sticker you attach to the broker. Ask: exactly once where, across which state transitions, and under which failures?

A practical implementation checklist

Before enabling retries on a mutating pipeline step, verify:

1. Operation identity: Can the caller express one logical operation with one stable key?
2. Key reuse: Is the same key persisted and reused across every retry and redelivery?
3. Storage invariant: Does a unique constraint enforce the rule at the final write boundary?
4. Parameter consistency: Does the system reject a reused key with different meaningful inputs?
5. Atomicity: Are the dedupe record and business mutation committed together?
6. External effects: Does each downstream call receive its own stable idempotency key?
7. Replay horizon: Does key retention exceed the longest realistic replay window?
8. Response behavior: Can a retry recover a semantically equivalent result rather than an ambiguous “already exists” error?
9. Observability: Can you count duplicates prevented, key conflicts, retries, and expired-key replays?
10. Testing: Have you injected failures after the mutation but before the acknowledgement?

That last test matters most. Happy-path unit tests rarely expose the dangerous window. Kill the worker after commit. Drop the response. Deliver the same message concurrently. Replay yesterday's batch. Then inspect both row counts and external effects.

What would make me wrong

Idempotency is not free, and not every operation needs the strongest contract.

If an operation is read-only, repeated execution may already be harmless. If the domain intentionally allows identical events, an aggressive deduplication key can delete legitimate work. If a duplicate can be reconciled cheaply and the cost of strict coordination is high, a simpler at-least-once design may be the better engineering choice.

The claim here is narrower: when one logical operation must create at most one durable side effect, retries are unsafe until identity, uniqueness, and atomicity agree on what “one” means.

Retries are how distributed systems recover from uncertainty. Idempotency is how they avoid turning that uncertainty into duplicate truth.

Sources

• AWS Builders' Library: Making retries safe with idempotent APIs
• Stripe API: Idempotent requests
• PostgreSQL: INSERT and ON CONFLICT
• Apache Kafka: Message delivery semantics