WAL & replicationadvancedFree in full

Measure and fence a Patroni failover

A three-node Patroni 4.1.3 cluster with three-node etcd and HAProxy completed graceful and crash failovers, rejected writes on a DCS-isolated former primary, and rejoined it as a replica. Timings are measured samples, not an SLO.

Problem

What you're actually looking at

The symptom as it shows up on a real server.

A topology diagram does not prove high availability. At 03:00 you need four facts you can show: exactly one elected writer, clients routed only to that writer, a measured gap from failure to first committed write through the real client endpoint, and proof that an isolated former primary cannot keep accepting writes. Missing any one of those turns failover into a hope.

The lab runs three PostgreSQL 16.14/Patroni nodes, three etcd members, and HAProxy. It measures one planned switchover, one hard leader kill, and a DCS-only network partition while the old database endpoint remains reachable. Use the same checks on a real cluster: REST topology, routed SQL, and a deliberate fencing test.

Simple terms

Failover is only real when applications can write again and the old primary cannot. Patroni uses the etcd leader lock to decide who may be writable. HAProxy follows Patroni's /primary endpoint. The lab times that chain and deliberately cuts only the old leader's DCS network to test split-brain protection. SQL path and lab harness verified; timings are host-specific samples.

Before you start

• Docker Compose with resources for 3 PostgreSQL/Patroni nodes, 3 etcd members, HAProxy, and a client.
• Patroni 4.1.3, PostgreSQL 16.14, etcd 3.5.17, and HAProxy 2.9 as pinned by docker/labs/ha-perf/ha.
• A throwaway environment. The compose credentials and REST exposure are lab-only.
• No Patroni/repmgr fork is required; the Patroni image is built from postgres:16 plus the pinned upstream Python package.

How to identify it

›Patroni /cluster (or patronictl list) shows exactly one Role=Leader with State=running, and every other member Role=Replica with State=streaming. Two Leaders is split-brain. Zero Leaders is a stuck election.
›The write VIP or HAProxy write port reaches only the node whose REST /primary returns HTTP 200. A 503 on /primary means that node must not receive writes.
›pg_is_in_recovery() is false through the client write endpoint and true on every replica. Timeline ID advances after a real failover (pg_controldata / pg_control_checkpoint).
›A hard leader failure restores commits through the same client DSN. Success signal: first committed write after the gap, with pg_is_in_recovery() still false on the new leader.
›A DCS-isolated former primary rejects a direct INSERT (read-only / not accepting commands) before it rejoins as a replica on the new timeline.

Pitfalls to avoid

✕Do not call these Docker timings a guaranteed RTO or a 99.999% SLO. Host load, ttl, loop_wait, DCS health, storage, DNS, and client retry behavior all matter.
✕Do not claim asynchronous Patroni guarantees zero data loss. maximum_lag_on_failover is an eligibility bound, not synchronous commit (see synchronous_commit and synchronous_standby_names in the PostgreSQL docs).
✕Do not run a single etcd member for an HA claim; this lab uses three members so one DCS failure keeps quorum.
✕Do not promote a standby without fencing the old primary or using a consensus-based manager. Dual writers are data divergence, not availability.
✕Do not treat HAProxy health as application retry logic; clients still need bounded reconnect and backoff, or they pin a dead connection through the whole outage window.
✕Do not switchover during a backlog of unreplayed WAL on the only eligible candidate; check pg_stat_replication lag and Patroni's lag eligibility first.

Trace it

01
Diagnose: measure graceful and crash failover
Time failure injection to first committed write through the real client endpoint, not through a direct connection to a known primary. The latest final-topology sample observed 41.412s from switchover trigger to a committed write and 42.160s after docker kill. Another local run observed 5.487s and 25.213s. That spread is the finding: one Docker result is not an SLO. Background: https://www.postgresql.org/docs/current/high-availability.html
Shell
```
cd docker/labs/ha-perf/ha
bash run-failover.sh
```
Measured sample · Patroni 4.1.3 · PostgreSQL 16.14
```
initial_leader=patroni1
switchover_candidate=patroni2
switchover_rto_ms=41412
crashed_leader=patroni2
crash_failover_rto_ms=42160
final_probe_rows=3
status=PASS
```
02
Diagnose: prove fencing with a DCS-only partition
Process death is not the only failure. Isolate only the leader's path to etcd while leaving the database port reachable. Success signals from this lab: /primary removed in 14.137s (leader_demote_ms), routed writes recovered in 29.066s (write_recovery_ms), isolated_write_rejected=true on a direct INSERT to the old leader, full_cluster_rejoined=true after heal. If isolated_write_rejected is false, you do not have fencing.
Shell
```
cd docker/labs/ha-perf/ha
bash run-fencing.sh
```
Measured fencing sample · Patroni 4.1.3
```
isolated_leader=patroni1
new_leader=patroni2
leader_demote_ms=14137
write_recovery_ms=29066
isolated_write_rejected=true
full_cluster_rejoined=true
status=PASS
```

Diagnose: verify one writer, streaming replicas, and route agreement

Patroni's cluster endpoint and the routed SQL endpoint must agree. Role/State columns: Leader/running is the only writer; Replica/streaming means WAL is applying; Replica/stopped or unknown needs investigation before the next drill. Through HAProxy write port, pg_is_in_recovery must be false.

Shell

curl -fsS http://localhost:56501/cluster
PGPASSWORD=postgres psql   -h localhost -p 55460 -U postgres -d postgres   -c "SELECT pg_is_in_recovery() AS must_be_false,
             current_setting('server_version') AS postgres_version,
             CASE WHEN NOT pg_is_in_recovery()
                  THEN pg_current_wal_lsn()::text
                  ELSE pg_last_wal_replay_lsn()::text
             END AS wal_position;"

Captured after former-primary rejoin

patroni1 | Replica | streaming
patroni2 | Leader  | running
patroni3 | Replica | streaming
pg_is_in_recovery = f

04
Diagnose: read the knobs that bound observed RTO
Patroni ttl (leader lock lifetime), loop_wait (how often members refresh), and retry_timeout dominate how fast a dead leader is noticed and replaced. Those live in patroni.yml, not in PostgreSQL GUCs. On the database side, confirm standbys are connected and whether any are synchronous before you trust a zero-loss story.
SQL
```
SELECT pg_is_in_recovery() AS this_node_is_standby;
SELECT application_name, state, sync_state,
       write_lag, flush_lag, replay_lag
FROM   pg_stat_replication
ORDER  BY application_name;
-- Expect on the leader: one row per streaming replica.
-- sync_state = sync only if synchronous_standby_names is set and matched.
```

Resolution approach

1.1. Baseline: curl /cluster (or patronictl list) and confirm exactly one Leader. Success: one Leader/running, others Replica/streaming.
2.2. Measure planned switchover to first committed write through the client write endpoint (lab: run-failover.sh). Success: switchover_rto_ms recorded and final_probe_rows increases.
3.3. Measure crash failover the same way (hard kill, not clean stop). Success: crash_failover_rto_ms recorded; new Leader appears; old node eventually rejoins as Replica.
4.4. Run a DCS-only partition (lab: run-fencing.sh). Success: isolated_write_rejected=true and write_recovery_ms recorded through the VIP.
5.5. After heal, confirm the former primary is a streaming replica on the new timeline and pg_is_in_recovery() is false only on the current Leader via the write VIP.
6.6. Record multiple samples under realistic load before publishing an internal RTO objective. Never promote a single lab number to an SLO.

Stop it recurring

01
Verify the routed writer after every drill
Run through the HAProxy write endpoint only. Success signal: must_be_false is f (false) and Patroni still shows a single Leader. If this returns true, the VIP is pointing at a replica.
SQL
```
SELECT pg_is_in_recovery() AS must_be_false,
       current_setting('server_version') AS postgres_version;
```
02
Confirm replicas are applying after rejoin
On the leader after the former primary rejoins. Success signal: every expected replica has state=streaming and replay_lag is stable, not climbing without bound.
SQL
```
SELECT application_name, client_addr, state, sync_state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_bytes_behind
FROM   pg_stat_replication
ORDER  BY application_name;
```

Verify you're done

SELECT pg_is_in_recovery() AS must_be_false;  -- run through the HAProxy write endpoint; expect f, and exactly one Patroni Leader

Last verified 2026-08-16 · PostgreSQL 16.14 + Patroni 4.1.3 (docker/labs/ha-perf/ha)

Related errors08006 55006

Related errors

SQLSTATEs this runbook resolves

The error pages that send an on-call engineer here.

Related runbooks

Continue the same incident path

Sibling procedures that cover the adjacent setup, recovery, or prevention step.

Other WAL & replication runbooks

Neighbouring incidents that share the same diagnostic surface.

All WAL & replication runbooks →

Connected

How this connects to the rest of the library

A live view of this page's real cross-references, what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.

Open in the interactive map →