WAL & replicationadvancedFree in full

Measure and fence a Patroni failover

A three-node Patroni 4.1.3 cluster with three-node etcd and HAProxy completed graceful and crash failovers, rejected writes on a DCS-isolated former primary, and rejoined it as a replica. Timings are measured samples, not an SLO.

Problem

What you're actually looking at

The symptom as it shows up on a real server.

A topology diagram does not prove high availability. You need one elected writer, routed client traffic, measured write recovery after switchover/crash, and a fencing test that proves an isolated former primary cannot keep accepting writes.

The lab runs three PostgreSQL 16.14/Patroni nodes, three etcd members, and HAProxy. It measures one planned switchover, one hard leader kill, and a DCS-only network partition while the old database endpoint remains reachable.

In plain English

Failover is only real when applications can write again and the old primary cannot. Patroni uses the etcd leader lock to decide who may be writable; HAProxy follows Patroni's /primary endpoint. The lab times that chain and deliberately cuts only the old leader's DCS network to test split-brain protection.

Before you start

  • • Docker Compose with resources for 3 PostgreSQL/Patroni nodes, 3 etcd members, HAProxy, and a client.
  • • Patroni 4.1.3, PostgreSQL 16.14, etcd 3.5.17, and HAProxy 2.9 as pinned by docker/labs/ha-perf/ha.
  • • A throwaway environment. The compose credentials and REST exposure are lab-only.
  • • No Patroni/repmgr fork is required; the Patroni image is built from postgres:16 plus the pinned upstream Python package.

How to identify it

  • ›Patroni reports exactly one Leader and two streaming Replicas.
  • ›HAProxy write traffic reaches only the node whose /primary endpoint returns 200.
  • ›A hard leader failure advances the timeline and restores writes through the same client endpoint.
  • ›A DCS-isolated former primary rejects a direct INSERT before it rejoins as a replica.

Pitfalls to avoid

  • ✕Do not call these Docker timings a guaranteed RTO or a 99.999% SLO. Host load, ttl, loop_wait, DCS health, storage, DNS, and client retry behavior all matter.
  • ✕Do not claim asynchronous Patroni guarantees zero data loss. maximum_lag_on_failover is an eligibility bound, not synchronous commit.
  • ✕Do not run a single etcd member for an HA claim; this lab uses three members so one DCS failure keeps quorum.
  • ✕Do not promote a standby without fencing the old primary or using a consensus-based manager — dual writers are data divergence, not availability.
  • ✕Do not treat HAProxy health as application retry logic; clients still need bounded reconnect/retry behavior.

Trace it

  1. 01

    Diagnose: measure graceful and crash failover

    The latest final-topology sample observed 41.412s from switchover trigger to a committed write and 42.160s after docker kill. Another local run observed 5.487s and 25.213s, demonstrating why one Docker result is not an SLO.

    Shell
    cd docker/labs/ha-perf/ha
bash run-failover.sh
    Measured sample · Patroni 4.1.3 · PostgreSQL 16.14
    initial_leader=patroni1
switchover_candidate=patroni2
switchover_rto_ms=41412
crashed_leader=patroni2
crash_failover_rto_ms=42160
final_probe_rows=3
status=PASS
  2. 02

    Diagnose: prove fencing with a DCS-only partition

    The old primary stayed reachable on the database network but lost etcd. Patroni removed /primary in 14.137s, HAProxy restored writes in 29.066s, and a direct INSERT on the former leader failed read-only.

    Shell
    cd docker/labs/ha-perf/ha
bash run-fencing.sh
    Measured fencing sample · Patroni 4.1.3
    isolated_leader=patroni1
new_leader=patroni2
leader_demote_ms=14137
write_recovery_ms=29066
isolated_write_rejected=true
full_cluster_rejoined=true
status=PASS
  3. 03

    Diagnose: verify one writer and two streaming replicas

    Patroni's cluster endpoint and the routed SQL endpoint must agree: one leader, two streaming replicas, and pg_is_in_recovery=false through HAProxy's writer port.

    Shell
    curl -fsS http://localhost:56501/cluster
PGPASSWORD=postgres psql   -h localhost -p 55460 -U postgres -d postgres   -c "SELECT pg_is_in_recovery(), current_setting('server_version');"
    Captured after former-primary rejoin
    patroni1 | Replica | streaming
patroni2 | Leader  | running
patroni3 | Replica | streaming
pg_is_in_recovery = f

Resolution approach

  1. 1.Measure failure injection to first committed write through the real client endpoint.
  2. 2.Test a DCS/network partition separately from process death.
  3. 3.Verify the former primary rejects writes and rejoins on the new timeline.
  4. 4.Record multiple samples under realistic load before setting an internal recovery objective.

Stop it recurring

  1. 01

    Verify the routed writer after every drill

    Run through the HAProxy write endpoint. A correct route returns pg_is_in_recovery = false and one Patroni leader.

    SQL
    SELECT pg_is_in_recovery() AS must_be_false,
       current_setting('server_version') AS postgres_version;

Verify you're done

SELECT pg_is_in_recovery() AS must_be_false;  -- run through the HAProxy write endpoint

Last verified 2026-08-16 · PostgreSQL 16.14 + Patroni 4.1.3 (docker/labs/ha-perf/ha)

Related errors0800655006

Related errors

SQLSTATEs this runbook resolves

The error pages that send an on-call engineer here.

More in this category

Other WAL & replication runbooks

Neighbouring incidents that share the same diagnostic surface.

All WAL & replication runbooks →

Connected

How this connects to the rest of the library

A live view of this page's real cross-references — what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.

Open in the interactive map →