Survive a connection storm when there is no pooler
A retry policy opened a fresh backend every time a query timed out, until every non-reserved slot in max_connections was gone. Triage from the reserved superuser slot, shed the connections that are holding but not working, then make the storm impossible.
Problem
What you're actually looking at
The symptom as it shows up on a real server.
Every PostgreSQL connection is a separate operating-system process, not a cheap handle. When an application answers slowness by retrying without pooling or backoff, each retry costs another process. Past a point the server spends its CPU forking and scheduling backends instead of running queries, so the database is technically up and effectively answering nothing.
A short latency blip on Meridian Freight's checkout path trips the application's retry logic. Each retry opens a new connection rather than reusing one, and thousands of concurrent users multiply that across every service. Within minutes the non-reserved slots are exhausted, new connections are refused with SQLSTATE 53300, and the on-call engineer cannot get a psql session in to look.
In plain English
PostgreSQL does not hand out lightweight connection handles — it starts a whole separate operating-system process for every connection. That is fine when an application borrows a few and gives them back. It falls apart when the application responds to slowness by opening more: every retry is another process, and past a point the server is busy creating and scheduling processes rather than answering queries. So load looks enormous while actual work goes to zero. The way out has two halves. Right now, get in through the slot PostgreSQL deliberately holds back for superusers, and close the connections that are sitting idle doing nothing. Afterwards, put a pooler in front, so that however hard the application retries it is sharing one small fixed set of server connections instead of creating new ones.
Full runbook for this incident
- The full identify checklist — the exact signals that tell you it's this incident
- Every diagnostic query; PostgreSQL 18 output is attached only to the steps we captured
- The resolution path and the pitfalls that make it worse
- Mitigation steps to stop it recurring, plus a verify-you're-done query
More in this category
Other Connections & auth runbooks
Neighbouring incidents that share the same diagnostic surface.
Connected
How this connects to the rest of the library
A live view of this page's real cross-references — what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.