Guarantee exactly-once settlements with idempotency keys
A retried payout request must not create a second settlement. A UNIQUE idempotency key plus INSERT … ON CONFLICT collapses a concurrent double-submit to one ledger row and returns the original id, so a carrier is paid exactly once even when the client retries.
Problem
What you're actually looking at
The symptom as it shows up on a real server.
At-least-once delivery is the default reality of the web: clients time out and resubmit, load balancers retry, and users double-click. Without a durable exactly-once boundary in the database, each retry of a payout becomes a second settlement row, a duplicate payment. Application-side dedupe checks race under concurrency; only a constraint the database enforces is safe.
Meridian Freight settles a delivered load. The client POSTs the payout, times out waiting for the response, and retries; two requests reach two backends at the same moment. The lab reproduces that race and shows how a UNIQUE (tenant_id, idempotency_key) constraint plus ON CONFLICT makes the second write a no-op that returns the first row's id.
Simple terms
Give every payout attempt a stable ‘idempotency key’ the client reuses on retry (for example load id + attempt number). Put a UNIQUE constraint on it. The first write wins; the second hits the constraint and, instead of inserting again, just returns the row that already exists. The database, not hopeful application code, is what guarantees one payout.
Full runbook for this incident
- The full identify checklist, the exact signals that tell you it's this incident
- Every diagnostic query; lab output is attached only to the steps we actually captured
- The resolution path and the pitfalls that make it worse
- Mitigation steps to stop it recurring, plus a verify-you're-done query
Card required. Cancel before day 7 and you are not charged.
Related runbooks
Continue the same incident path
Sibling procedures that cover the adjacent setup, recovery, or prevention step.
More in this category
Other Mission-critical platform runbooks
Neighbouring incidents that share the same diagnostic surface.
Connected
How this connects to the rest of the library
A live view of this page's real cross-references, what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.
Need the full procedure?
Pro runbooks finish the incident path
Free runbooks teach the shape. Pro opens the full step transcript, edge cases, and prevention depth.