The Myth of Exactly Once

It is 11:58 p.m., and somewhere a man is staring at his phone, watching a spinner that refuses to stop spinning.

He has just bought a concert ticket. He tapped Pay, the screen dimmed, and then — nothing. The little wheel turns. His thumb hovers. The Wi-Fi icon flickers. Did it go through? He has no way to know, and the silence is unbearable, so he does what every human being does in this situation: he taps Pay again.

Two seconds later his phone buzzes twice. Two confirmation emails. Two charges. One very annoyed customer. And somewhere behind the scenes, an engineer about to learn the single most important truth in distributed systems. The lucky ones learned it long ago.

You cannot do things exactly once. You can only make it look like you did.

The wish

Every system that touches money, or messages, or inventory, or anything a human will notice, wants the same thing. When the user says do this, the system should do it. Once. Not zero times, which loses the sale. Not twice, which loses the customer. Exactly once.

It is such a reasonable wish that we rarely examine it. "Exactly-once delivery" appears on marketing pages for message queues and payment processors as if it were a checkbox, a feature you switch on. It is printed with the quiet confidence of a law of physics.

It is not a law of physics. It is closer to a magic trick. And like every good magic trick, understanding how it is performed is far more interesting than believing it is real.

Why the network will always betray you

Here is the uncomfortable foundation. When your phone sends a request to a server, two things can go wrong, and crucially, they are indistinguishable from where you are standing.

The request might never arrive. Or the request might arrive, be processed perfectly, and the response might be the thing that vanishes on the way back. From the client's point of view these two scenarios look identical: you sent something, and you heard nothing. The spinner spins either way.

This is not a flaw in your code. It is the shape of the world. Computer scientists named it decades ago, the Two Generals Problem, and dressed it up in a small fable.

Two armies camp on opposite hills, a hostile valley between them. To win, they must attack at the same moment. Their only way to coordinate is to send a messenger running through the valley, where messengers are sometimes captured. General A sends word: attack at dawn. But did it arrive? She needs a confirmation. The messenger returns: confirmed. But did that arrive? General B now needs a confirmation of the confirmation. And so on, forever. No finite number of messengers can ever make both generals certain.

The valley is your network. The messengers are your packets. And the grim, beautiful conclusion is that two parties communicating over an unreliable channel can never achieve common knowledge with certainty. There is always one more acknowledgement you'd want and cannot guarantee.

So when the man's spinner spins, the system genuinely does not know whether he has paid. Not because it was written badly, but because certainty is not on the menu.

The two honest answers

If you cannot have exactly-once, what can you have? There are exactly two honest delivery guarantees, and every real system is built on one of them.

The first is at-most-once. You try the operation. If anything goes wrong (a timeout, a dropped connection, a server that fell over mid-request), you simply give up. You never retry. This guarantees you will never do the thing twice, because you will sometimes not do it at all. It is the guarantee of someone who would rather drop a sale than risk a double charge. Fast, simple, and quietly lossy.

The second is at-least-once. You try the operation, and if you are unsure whether it succeeded, you try again. And again, if you have to. This guarantees the thing will eventually happen. But it might happen more than once, because some of those "failed" attempts were actually successes whose acknowledgements got lost in the valley.

That is the whole landscape. At-most-once loses things. At-least-once duplicates things. There is no third door marked exactly-once you can walk through.

And yet your bank does not double-charge you every time your connection hiccups. So what gives?

The trick: stop fighting duplicates, start absorbing them

Here is the pivot, and it is one of those ideas that quietly rearranges how you think about everything afterward.

The industry made a choice. It picked at-least-once, because losing a payment is far worse than the alternative, and then it neutralised the downside. It did not try to prevent duplicates. Networks make that impossible, remember. Instead it made duplicates harmless.

The word for "harmless to repeat" is idempotent.

An operation is idempotent if doing it five times leaves the world in exactly the same state as doing it once. Turning on a light switch that is already on. Setting your profile name to "Ada" when it is already "Ada." Deleting a file that is already gone. The second, third, and tenth attempts simply have nothing left to do.

"Exactly-once" is not a delivery guarantee at all. It is the combination of an at-least-once channel that keeps trying, plus an idempotent receiver that yawns at repeats. The messengers may run the valley a hundred times. The order is still given once.

This reframing is the entire game:

• You will not stop the duplicate request from arriving.
• You will make sure the second one changes nothing.

Once that clicks, the engineering becomes almost calm.

How it is actually built

So how do you make charging a credit card, about as non-idempotent an act as exists, behave like flipping a switch? You give every intention a name, and you remember the names you've already honoured.

That name is an idempotency key: a unique token the client generates once, before the first attempt, and attaches to every retry of the same logical operation. When our anxious customer taps Pay and then taps it again, both requests carry the same key, because they are the same intention, not two intentions that happen to look alike.

The server's job is now refreshingly small:

async function charge(req: ChargeRequest) {
  const key = req.idempotencyKey;
 
  // Have we seen this intention before?
  const existing = await db.idempotency.findByKey(key);
  if (existing) {
    // Yes: return the original outcome. Do NOT charge again.
    return existing.response;
  }
 
  // No: claim the key and do the work, atomically.
  return db.transaction(async (tx) => {
    // INSERT with a UNIQUE constraint on `key`.
    // If a concurrent retry beat us here, this throws, and we
    // fall back to returning the winner's result.
    await tx.idempotency.insert({ key, status: "pending" });
 
    const result = await paymentProvider.charge(req.amount, req.card);
 
    await tx.idempotency.update(key, {
      status: "done",
      response: result,
    });
 
    return result;
  });
}

Read that transaction slowly, because every line is load-bearing.

The unique constraint on the key is doing the real work. It turns "did someone already start this?" from a question you ask (read, then decide, then write, with a gap a duplicate can slip through) into a fact the database enforces. Two simultaneous retries race to insert the same key; the database lets exactly one win and rejects the other. The loser does not charge the card. It simply waits and returns what the winner produced.

The customer taps Pay twice. Two requests arrive carrying one key. One does the charging. The other discovers the work is already claimed and hands back the same receipt. Two taps, two responses, one charge. The spinner lied; the ledger did not.

Where it gets genuinely hard

If the story ended there, exactly-once would be a footnote, not a career. It does not end there, and the places it frays are where good engineers earn their keep.

The work itself must be atomic with the bookkeeping. Notice that the example records the result inside the same transaction that does the work. If you charge the card first and then crash before writing "done," your next retry sees no record and charges again. The intention-log and the side effect have to commit together, or you have rebuilt the very bug you set out to kill.

Some side effects refuse to be transactional. A database row is easy to roll back. An email is not. Once you have called the third-party that physically sends a message, or moved real value on a blockchain you do not control, there is no rollback. For these, you protect the boundary: check "have I already sent this?" immediately before the irreversible act, and record "I sent it" immediately after, as close together as the laws of your infrastructure allow. You are shrinking the window of doom, not eliminating it. Being honest about that distinction is the difference between a robust system and a smug one.

Keys have to expire, and expiry is a trap. You cannot store every idempotency key until the heat death of the universe. So you keep them for a window: twenty-four hours, seven days, whatever your retry horizon demands. But pick the window too short and a slow, legitimate retry arrives after you have forgotten its key, and is treated as brand new. The duplicate you so carefully designed against walks right through the front door because you swept the floor too early.

Exactly-once is a property of the whole path, not a component. A queue can promise "exactly-once processing," but it is really doing the same trick one layer down, deduplicating on a message ID inside its own window, and the instant your handler talks to a system outside that boundary, the guarantee stops at the door. There is no global setting. There is only a chain of idempotent steps, each one honest about where its protection ends.

What this teaches you about everything else

I keep coming back to this topic, long after the payments work that first forced me to understand it, because the lesson generalises so cleanly it is almost suspicious.

You will not prevent the messy, repeated, out-of-order reality of a distributed world. The retries will come. The duplicates will come. The acknowledgement you were counting on will dissolve in the valley at the worst possible moment. The senior move is not to build a wall high enough to stop all of it. That wall cannot be built. The senior move is to design the receiver so that the mess lands softly, so that the second arrival has nothing left to do.

Make the operation idempotent, and "how many times did this happen?" stops being a question that can hurt you.

"Exactly once" was never a delivery guarantee. It is a feeling: the feeling a user gets when they tap Pay twice in a panic and are charged once anyway. Behind that feeling is no magic and no law of physics. There is only a unique constraint, a transaction drawn tightly around a side effect, and an engineer who understood that the goal was never to stop the duplicate from arriving.

The goal was to make sure it didn't matter when it did.