Idempotency Is a Design Decision, Not a Retry Policy

Every event-driven system eventually delivers the same message twice. The broker redelivers after a rebalance, a consumer crashes between processing and committing its offset, or an upstream service retries a call it never got an answer to. None of these are bugs. They are the normal operating conditions of an at-least-once system.

The mistake I see most often is treating this as a retry problem — tuning backoff, adding a dead-letter queue, capping attempts — when it is really a data modelling problem.

What "idempotent" actually has to mean

A handler is idempotent when processing the same logical event n times leaves the system in the same state as processing it once. That is a statement about your database, not about your consumer code.

Concretely, that usually means one of three things:

1. A natural unique key. The event carries an identifier that maps to a unique constraint in your store. A second insert fails loudly and you swallow the conflict.
2. A processed-events ledger. You record (event_id, consumer_group) in the same transaction as the side effect. If the row already exists, you skip.
3. A state machine with guarded transitions. UPDATE payouts SET status = 'settled' WHERE id = $1 AND status = 'pending' affects zero rows the second time.

All three share one property: the guard and the side effect commit together. If you write the ledger row in Redis and the side effect in MySQL, you have not made the operation idempotent — you have added a second thing that can fail independently.

The batch case is where it gets interesting

Single-event idempotency is well-trodden. Batch processing is where I have watched systems quietly drift.

A compensation run that processes a thousand records per batch has a partial-failure surface that a single-event handler does not. If record 743 throws, you have already committed 742 side effects. Retrying the batch from the top will double-apply them unless every record inside carries its own guard.

The fix is not to wrap the batch in one giant transaction — that locks too much and times out at scale. It is to make the batch a loop of independently idempotent operations, and to make the batch itself resumable by recording per-record progress. The batch becomes a scheduling concern; correctness stays at the record level.

Reconciliation is the honest part

Even with all of this, I run reconciliation. Not because I expect the guards to fail, but because "I expect it to work" is not an operational stance.

A reconciliation job answers a narrow question: for a given window, does the set of events the source emitted match the set of side effects we recorded? Gap detection over a monotonic sequence catches missed events. Sum checks over financial totals catch double-applied ones. Both should produce a backfill plan rather than fixing things silently, so a human sees the delta before it disappears.

The systems I trust are not the ones that never lose an event. They are the ones that can tell me, within an hour, exactly which event they lost.