Stop work_mem from OOM-killing the server
work_mem is a per-operation budget, not a per-server one. Multiply it by plan nodes, by concurrency and by hash_mem_multiplier before you raise it — then give the memory to the few queries that need it instead of to every backend at once.
Problem
What you're actually looking at
The symptom as it shows up on a real server.
PostgreSQL memory is not only global. work_mem is granted per sort, hash and materialise node, per query, per session, and there is no cluster-wide ceiling that stops the total. A value chosen to make one report faster becomes an unbounded multiplication under concurrency, and the kernel resolves it by killing the largest process — which is PostgreSQL.
Meridian raised work_mem globally to speed up a nightly report. It worked. Weeks later a burst of concurrent reporting queries each claimed several sort and hash nodes' worth of that budget at the same moment. Average memory graphs stayed calm because the spike was short and spread across processes; the server still ran out of RAM and came back through crash recovery.
In plain English
work_mem is not 'how much memory the database may use for sorting'. It is 'how much memory each sort or hash step in each query may use, at the same time'. A single query with several sorts and joins can claim it several times over, and every concurrent session can do the same again. So the number you typed into the config file gets multiplied by two things you did not type: how many memory-hungry steps a plan happens to have, and how many sessions are running one at once. That is why a value that looked safe on a quiet evening can add up to more memory than the machine has during a busy hour. When Linux runs out, it does not slow down politely — it kills the largest process, and the largest process is PostgreSQL.
Full runbook for this incident
- The full identify checklist — the exact signals that tell you it's this incident
- Every diagnostic query; PostgreSQL 18 output is attached only to the steps we captured
- The resolution path and the pitfalls that make it worse
- Mitigation steps to stop it recurring, plus a verify-you're-done query