The Read-Then-Write Balance Trap That Records History That Never Happened

The trap: a balance that reconciles to nothing

Here’s the failure mode that should worry any operator running a ledger, an inventory count, or any shared counter: two operations land on the same account at the same moment, both succeed, no error is thrown — and one of them silently vanishes. Worse, your audit trail confidently records a running balance that the account never actually held. Nothing looks broken until someone tries to reconcile the numbers and finds they don’t add up.

We hit this in the ledger of our internal OperationsHub. It’s the kind of bug that hides in plain sight because every individual write is correct. The problem only exists in the gap between reading a value and writing it back.

Why read-then-write loses updates

Several services wrote to account balances — expense, payment, invoice, transaction, and account transfer. Each did the intuitive thing:

1. Read the account’s current balance in application code.
2. Compute the new balance in JavaScript (balance - amount).
3. Write that new absolute number back.

Run two of those concurrently against the same account and you get a classic lost-update race — where two writers each start from the same stale snapshot, so the second write overwrites the first instead of building on it. Say the balance is 100 and two postings of -10 arrive together. Both read 100. Both compute 90. Both write 90. The account ends at 90 when it should be at 80. One posting is simply gone from the balance, even though its ledger row still exists.

The second, quieter problem is the one that breaks trust. Each ledger row stamped its balanceAfter — the running balance recorded alongside a transaction — from that same computed-in-JS value. So the audit trail didn’t just lose a number; it recorded a balance that never existed at any point in the account’s real history. That’s the field auditors and reconciliation jobs lean on, and it was fiction.

Why raising the isolation level wouldn’t have saved us

The reflex fix is “wrap it in a transaction, bump the isolation level.” That reflex is wrong here, and this is the part worth internalising.

Some of these writes happened entirely outside a database transaction. Isolation levels govern how transactions see each other — they do nothing for a bare read followed by a bare write. And even inside a transaction, a read-then-write of an absolute value is still computing from a snapshot the application captured earlier. You can escalate isolation until writes start deadlocking and serialisation-failing, and you’ve traded a silent bug for a loud one, not fixed the shape of the operation.

The operation itself is the problem: computing an absolute result from a value you read a moment ago.

The fix: let the database do the arithmetic

The change was to stop sending absolute balances and start sending relative ones:

• Every site now uses atomic relative operations — increment / decrement — evaluated by the database itself, so balance - 10 is computed against the live row at write time, not against an application snapshot. Concurrent postings compose correctly.
• balanceAfter is now read from the row the update actually returned, not from the value we computed beforehand. The audit trail can only record balances that really occurred.
• Balance and ledger writes are wrapped in a single transaction wherever they weren’t already, so a row never records a balance its account didn’t reach.

While we were in there, transfer() had a duplicated internal-transfer code path that rebuilt its own near-identical operations array instead of reusing the one already constructed — exactly the kind of copy that drifts out of sync over time. That’s now a single path, and it rejects zero, negative, and self-transfers outright.

We verified it with a concurrency test that fires 10 simultaneous writes at one account and asserts that all 10 balanceAfter values come back distinct. Under the old code, collisions were the whole point of the bug.

The transferable rule

If you take one thing from this: read-then-write on a shared numeric field is a lost-update trap regardless of transactions or isolation level, whenever the write computes an absolute value from an earlier read. Balances, inventory counts, sequence numbers, rate-limit counters — same shape, same risk. Use the database’s atomic relative operations instead.

And derive any audit or history field from what the write actually returned, never from the value you computed before it. Otherwise your history can faithfully record events that never happened — which is worse than a wrong number, because it looks like proof.