Measure what a failover costs your application
Timing a promotion measures the database's opinion of itself. Measuring from a client that must reconnect and re-route gave 9.187 s on one run and 4.500 s on the next, and the difference was almost entirely how long the promotion took to start.
Problem
What you're actually looking at
The symptom as it shows up on a real server.
Failover time is usually quoted from the server side: how long promotion took. That number leaves out everything the application experiences — the dead connection, the reconnect, the discovery of which node is now writable. The result is an HA claim nobody can defend, because the thing that was measured is not the thing that was lost.
Meridian wants a defensible failover number before an availability review. Rather than timing pg_ctl promote, they keep a writer running throughout and record every attempt it makes.
In plain English
If you time how long the database took to promote, you are asking the database how it felt about the outage. Your users were somewhere else: waiting on a connection that had already died, then waiting for the app to notice, reconnect, and find the machine that now accepts writes. So the honest way to measure a failover is to keep something writing the whole time and count the gap between the last write that worked and the next one that worked. That gap is the real bill.
Full runbook for this incident
- The full identify checklist — the exact signals that tell you it's this incident
- Every diagnostic query; PostgreSQL 18 output is attached only to the steps we captured
- The resolution path and the pitfalls that make it worse
- Mitigation steps to stop it recurring, plus a verify-you're-done query
More in this category
Other High availability runbooks
Neighbouring incidents that share the same diagnostic surface.
Connected
How this connects to the rest of the library
A live view of this page's real cross-references — what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.
Fixes these errors