Connections & authadvancedFree in full

Survive a connection storm when there is no pooler

A retry policy opened a fresh backend every time a query timed out, until every non-reserved slot in max_connections was gone. Triage from the reserved superuser slot, shed the connections that are holding but not working, then make the storm impossible.

Problem

What you're actually looking at

The symptom as it shows up on a real server.

Every PostgreSQL connection is a separate operating-system process, not a cheap handle. When an application answers slowness by retrying without pooling or backoff, each retry costs another process. Past a point the server spends its CPU forking and scheduling backends instead of running queries, so the database is technically up and effectively answering nothing.

A short latency blip on Meridian Freight's checkout path trips the application's retry logic. Each retry opens a new connection rather than reusing one, and thousands of concurrent users multiply that across every service. Within minutes the non-reserved slots are exhausted, new connections are refused with SQLSTATE 53300, and the on-call engineer cannot get a psql session in to look.

Simple terms

PostgreSQL does not hand out lightweight connection handles. It starts a whole separate operating-system process for every connection. That is fine when an application borrows a few and gives them back. It falls apart when the application responds to slowness by opening more: every retry is another process, and past a point the server is busy creating and scheduling processes rather than answering queries. So load looks enormous while actual work goes to zero. The way out has two halves. Right now, get in through the slot PostgreSQL deliberately holds back for superusers, and close the connections that are sitting idle doing nothing. Afterwards, put a pooler in front, so that however hard the application retries it is sharing one small fixed set of server connections instead of creating new ones.

Before you start

• A superuser role, or one granted pg_signal_backend, which can still connect through the reserved slots when ordinary roles are refused.
• Agreement on which application_name values are safe to terminate before you terminate anything.

How to identify it

›New connections fail with SQLSTATE 53300 (too many connections) while sessions that already hold a slot keep running.
›pg_stat_activity is dominated by backends in state idle rather than active: the slots are held, not working. idle in transaction is worse because those also hold locks and pin a snapshot.
›Connection count climbed far faster than query volume did, which points at retries rather than real traffic. The backend_start histogram spikes in the last few minutes.
›CPU is high while throughput is flat, because the work is process handling and context switching rather than query execution.
›Census query: in_use is at or above max_connections - superuser_reserved_connections - reserved_connections, so only the reserved pool can still connect.

Pitfalls to avoid

✕Do not raise max_connections as the first move. It needs a restart you cannot afford mid-incident, and more backends competing for the same RAM makes an out-of-memory kill more likely, not less.
✕Do not terminate backends indiscriminately; killing an active backend rolls back real work and, against a retrying client, immediately buys you a replacement connection.
✕Do not assume your language's connection pool protects the server. HikariCP, ADO.NET, database/sql and SQLAlchemy all cap reuse inside one process, so N replicas each holding a modest pool still multiply into a server-side total nobody bounded.
✕Do not reach for session pooling to absorb a storm. It holds one server connection per client for the whole session, so it barely multiplexes. Transaction pooling is what collapses many clients onto few backends; keep session pooling for the ETL and BI workloads that genuinely need session state, temp tables or prepared statements.
✕Do not leave the retry policy uncapped once a pooler is in place; a pooler in front of an unbounded retry loop just moves the queue one layer out.
✕Do not set idle_session_timeout on the server connections owned by PgBouncer or another pooler; the pooler expects those backends to stay idle between client checkouts.

Trace it

01
Diagnose: get in through the reserved slot
PostgreSQL always holds superuser_reserved_connections back from ordinary roles, which is why a superuser can still connect when everyone else is refused. PostgreSQL 16 added reserved_connections for roles granted pg_use_reserved_connections. Columns: max_connections is the hard ceiling of client backends; superuser_reserved and reserved are subtracted from what non-privileged roles may use; in_use is live client backends. Docs: https://www.postgresql.org/docs/current/runtime-config-connection.html
SQL
```
SELECT (SELECT setting::int FROM pg_settings WHERE name = 'max_connections')                AS max_connections,
       (SELECT setting::int FROM pg_settings WHERE name = 'superuser_reserved_connections') AS superuser_reserved,
       (SELECT setting::int FROM pg_settings WHERE name = 'reserved_connections')           AS reserved,
       (SELECT count(*) FROM pg_stat_activity WHERE backend_type = 'client backend')        AS in_use;
```
02
Diagnose: separate working backends from slots that are only holding
This is the split that decides what is safe to touch. state=active is real work (leave alone). state=idle is a checked-out slot doing nothing. state=idle in transaction holds locks and a snapshot as well as a slot. oldest_in_state shows how long the worst offender has sat there.
SQL
```
SELECT state,
       count(*)                        AS backends,
       max(now() - state_change)       AS oldest_in_state
FROM   pg_stat_activity
WHERE  backend_type = 'client backend'
  AND  pid <> pg_backend_pid()
GROUP  BY state
ORDER  BY backends DESC;
```
03
Diagnose: attribute connections to a caller
A storm almost never comes from everywhere at once. It comes from the one service whose retry loop is unbounded. Grouping by application_name, client_addr and usename names the culprit and gives you the allowlist before terminating anything. If application_name is unset for most rows, that is itself the finding: set it in the connection string so the next incident is attributable in seconds.
SQL
```
SELECT coalesce(nullif(application_name, ''), '(unset)') AS application_name,
       client_addr,
       usename,
       state,
       count(*)                  AS conns,
       max(now() - state_change) AS oldest
FROM   pg_stat_activity
WHERE  backend_type = 'client backend'
  AND  pid <> pg_backend_pid()
GROUP  BY 1, 2, 3, 4
ORDER  BY conns DESC;
```
04
Diagnose: separate a retry storm from steady saturation
This is the check that changes the fix, and almost nobody runs it. backend_start is when each backend was forked, so bucketing by minute turns the connection list into a churn histogram. A flat profile of old backends means you are simply sized too small: raise the pool or the ceiling carefully. A spike of very recent backends means connections are being created faster than they are being released, which is a retry storm, and raising max_connections there just gives the storm more room to grow.
SQL
```
SELECT date_trunc('minute', backend_start) AS started_minute,
       count(*)                                AS connections_opened
FROM   pg_stat_activity
WHERE  backend_type = 'client backend'
  AND  backend_start > now() - interval '30 minutes'
GROUP  BY 1
ORDER  BY 1 DESC;
```

Resolution approach

1.1. Connect as a superuser (reserved slots) and run the census. Success: you have a session while ordinary roles still see 53300.
2.2. Read the churn histogram before touching any limit. Spike of new backend_start => retry storm. Flat old backends => capacity problem.
3.3. Shed safest slots first: state=idle older than your threshold, scoped by application_name. Success: in_use drops and new app connections succeed without 53300.
4.4. Then consider idle in transaction past a short threshold (they hold locks). Never mass-kill state=active.
5.5. Cap the application's own pool size and add capped exponential backoff with jitter so a latency blip cannot multiply into a storm.
6.6. Put a transaction-mode pooler in front, sized so pool + reserve stays below max_connections. Success: verificationQuery shows total well below max_connections and stable under load.

Stop it recurring

01
Terminate idle slots, oldest first, by application
Run the SELECT first and read it, then swap in pg_terminate_backend once you agree with the list. Scoping to state = idle and an age threshold keeps working sessions untouched, and naming the application keeps the blast radius to the service that caused it. pg_terminate_backend needs superuser or membership in pg_signal_backend. Docs: https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL
SQL
```
SELECT pid, usename, application_name, state, now() - state_change AS idle_for
--     , pg_terminate_backend(pid)          -- uncomment once the list above looks right
FROM   pg_stat_activity
WHERE  backend_type = 'client backend'
  AND  pid <> pg_backend_pid()
  AND  state = 'idle'
  AND  state_change < now() - interval '10 minutes'
ORDER  BY state_change;
```
02
Let the server defend its own slots
Two timeouts turn slot leaks from an incident into a non-event. idle_in_transaction_session_timeout ends sessions holding a transaction open with no query in flight, which also releases their locks and snapshot. idle_session_timeout (PostgreSQL 14+) ends sessions idle outside a transaction. Set it only where clients reconnect cleanly; never for a pooler's own server connections. Both are reloadable (context=user).
SQL
```
ALTER SYSTEM SET idle_in_transaction_session_timeout = '60s';
ALTER SYSTEM SET idle_session_timeout = '30min';
SELECT pg_reload_conf();

SELECT name, setting, unit, context
FROM   pg_settings
WHERE  name IN ('idle_in_transaction_session_timeout', 'idle_session_timeout');
```

Verify you're done

SELECT (SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections,
       count(*) FILTER (WHERE state = 'active') AS active,
       count(*) FILTER (WHERE state = 'idle')   AS idle,
       count(*)                                 AS total
FROM   pg_stat_activity WHERE backend_type = 'client backend';  -- total should sit well below max_connections and stay flat

Last verified 2026-08-15 · PostgreSQL 17.11 (SQL path verified; no fabricated step output)

Related errors53300 55P02 25P03

Related errors

SQLSTATEs this runbook resolves

The error pages that send an on-call engineer here.

Related runbooks

Continue the same incident path

Sibling procedures that cover the adjacent setup, recovery, or prevention step.

Diagnose PgBouncer queueing and session-state leaksFree

Other Connections & auth runbooks

Neighbouring incidents that share the same diagnostic surface.

Rotate md5 passwords to SCRAM without locking out clients

Browse all 105 runbooks →

Connected

How this connects to the rest of the library

A live view of this page's real cross-references, what explains it, what fixes it, what to tune, and where to go next. Every link is an authored relationship, not a guess.

Open in the interactive map →