Interview Prep Zoneby CuriouserLabs

Question 4 of 5 · senior level

Is exactly-once delivery possible? Design a payment consumer that never double-charges.

🎤 Say this first

*Exactly-once delivery is impossible — the Two Generals result means the sender can never know whether a lost acknowledgement meant 'not delivered' or 'delivered, ack lost', so it must either retry (at-least-once, risking duplicates) or not (at-most-once, risking loss). What is achievable is exactly-once processing: at-least-once delivery plus idempotent consumers. The production recipe: the producer sends a stable idempotency key (business identifier, not a random per-attempt UUID); the consumer records that key in the same transaction as the side effect, using a uniqueness constraint so a duplicate fails atomically; external effects that can't join the transaction (charging a card, sending an email) are made idempotent by passing the key downstream — every serious payment API accepts one. 'Kafka exactly-once' is this same idea implemented inside Kafka: idempotent producers plus transactional offset commits, valid only within* Kafka.

The full picture

Idempotent consumer: the dedupe record and the effect commit together
@Transactional
public void handle(PaymentRequested event) {
    // The key must be stable across retries: derive it from the business event,
    // never from the delivery attempt.
    String key = event.paymentId();

    try {
        processedRepo.insert(key, Instant.now());   // UNIQUE constraint on key
    } catch (DuplicateKeyException dup) {
        return;                                     // already handled — drop it
    }

    ledger.debit(event.accountId(), event.amount());  // same transaction as the marker
    gateway.charge(event.amount(), key);              // key -> gateway dedupes too
}
// The uniqueness constraint is what makes this safe under CONCURRENT duplicates,
// not just sequential ones: a "SELECT then INSERT" check has a race window and
// two consumers can both pass it.
SemanticMechanismRiskFit
At-most-oncefire and forget; ack before processingmessage lossmetrics, logs, telemetry
At-least-onceretry until acked; ack after processingduplicatesthe default — pair with idempotency
Exactly-once processingat-least-once + dedupe/idempotencystorage for keyspayments, orders, anything with side effects
Kafka EOSidempotent producer + transactionsKafka-internal onlystream processing read→process→write
  • Ack timing decides the semantic, and it's a one-line change: ack before processing = at-most-once; ack after = at-least-once. Say which one your framework does by default (Spring's AckMode, SQS visibility timeout, Kafka's enable.auto.commit) — teams are frequently surprised by their own configuration.
  • The transactional outbox pattern is the answer to 'how do I update the database and publish an event atomically': write the event to an outbox table in the same local transaction, and let a relay (or CDC via Debezium) publish it. This avoids the dual-write problem, where a crash between the DB commit and the broker publish loses the event or invents one.
  • Dedupe storage needs a retention policy — you can't keep idempotency keys forever. Pick a window comfortably longer than your maximum retry/redelivery horizon (24h–7d is typical), index by key, and TTL the rest. Unbounded dedupe tables are a slow-burning operational problem.
  • *Idempotency is not only about duplicates — it's about reordering too.* A retry can arrive after a subsequent update. Guard with a version/sequence check (WHERE version = ?) so a stale replay can't overwrite newer state; otherwise your consumer is idempotent but not order-safe.
  • Be precise about Kafka's exactly-once: it covers consume→transform→produce within Kafka, atomically committing offsets and output records. The moment your consumer calls an external API or another database, you're back to at-least-once and idempotency keys. Overstating Kafka EOS is a common and easily-caught error.

🔄 Likely follow-up questions

  • Why must the idempotency key be stable across retries rather than per-attempt?
  • How does the transactional outbox solve the dual-write problem?
  • What breaks if a duplicate arrives after a later update to the same record?