seniorDatabase SRE / Customer ReliabilityPro answerTested in a controlled lab, summarised

A customer says “the database is down.” In five minutes, how do you separate a broken connection path, connection saturation, wrong-node routing, lock waits, and a query-specific failure—and what do you communicate before you know root cause?

The database is down is a report, not a diagnosis, and this question is really about how quickly you can convert one into the other without breaking anything. Interviewers use it because it exposes two habits at once. The first is whether you test the path the customer is actually using or the one that is convenient for you. The second is whether you can read connection state accurately, because the most common wrong turn here is counting every server process as a client and declaring saturation that does not exist. The communication half is scored just as hard: panels want to hear what you would say while you still do not know. This page covers ordering the checks and the update that goes out first.

What the interviewer is scoring

Weak answer: counts every PostgreSQL process as a client connection, treats “waiting” as a session state, or fails over because one pool or one query is stuck.

In short: Test the path the customer actually uses, then bound the blast radius. A successful trivial transaction proves only that tested path and transaction; it does not prove that the workload is healthy.

ProFrom a real interview loop

The spoken answer and the reasoning behind it

The question, the intro, what the interviewer is scoring, the reasoning outline, what you would verify and the follow-up probes are all open above. Pro unlocks the spoken answer to “A customer says “the database is down.” In five minutes, how do you separate a broken connection path, connection saturation, wrong-node routing, lock waits, and a query-specific failure—and what do you communicate before you know root cause?”, the reasoning behind it, and the verification that backs it.
  • The full 90-second answer, written first person, the way you would actually say it
  • How I reason through it: the mechanism, the decision points, and where the claim stops
  • Verification steps with a written summary of what a controlled lab run showed, labelled as a summary rather than a transcript.
  • Worked responses to the 3 follow-up probes listed above, plus the traps that lose the point

Card required. Cancel before day 7 and you are not charged.

Compare plans

How to reason through it

  • Reproduce from the application network with the application endpoint, role, database, TLS mode, and a timed trivial transaction.
  • On a reachable server, count only backend_type='client backend' by state; do not mix autovacuum, WAL sender, background writer, or other backend types into client capacity.
  • Inspect wait_event_type and wait_event separately from state; compare ordinary-role capacity with max_connections - reserved_connections - superuser_reserved_connections.
  • Corroborate saturation with failed-connection evidence and the pool’s queue depth, checkout latency, timeouts, and configured limit; identify lock blockers with pg_blocking_pids(pid) and pg_locks.
  • Use pg_is_in_recovery() only to identify recovery state, not to declare health; split the blast radius by pool/direct path, read/write path, tenant, zone, and region.
  • Communicate observed symptom, confirmed impact, current blast radius, the next discriminating check, unsafe actions being withheld, and the next update time.

What I would verify

  • Time a connection and a trivial transaction from the application network, not from the database host.
  • Group client backends by state, and read wait events as a separate dimension rather than as a state.
  • Compare usable capacity for an ordinary role against the ceiling minus both reserved classes.
  • Confirm a lock theory with the blocking-PID relationship before cancelling anything.
  • Split impact by pooled versus direct, read versus write, tenant, and zone before calling it total.

Follow-ups they push on

  • You connect fine as an admin. Is the database healthy?
  • The pool is at its limit and the server is not. What do you do?
  • What do you tell the customer at minute four?

The probes are open. Pro carries the spoken answer, the reasoning behind it, and the verification steps, including a worked response to each of these.

Concepts tested

incident triageclient connection capacitywait eventslock blockingblast radiuscustomer communicationfailover guardrails

Learn it, run it, then say it

Three steps, in order. Nothing here is a detour.

Questions that go with this one