pg_control File Corrupt

Scenario

After a storage failure, PostgreSQL fails to start with FATAL: could not read from file "global/pg_control": Input/output error. The DBA finds the pg_control file is 0 bytes or shows filesystem errors. pg_control stores critical cluster state: the database system identifier, checkpoint location, WAL positions, and cluster state. Without a readable pg_control, PostgreSQL cannot start.

How to Identify

Conditions:

• PostgreSQL fails to start with could not read from file "global/pg_control"
• pg_controldata $PGDATA fails or shows garbage
• global/pg_control is zero bytes or corrupted
• Storage error occurred (disk failure, RAID failure, filesystem corruption)
• File may be intact but unreadable due to filesystem mount issues

Analysis Steps

-- pg_controldata shows current state (run from OS, not SQL):
-- pg_controldata $PGDATA
-- Look for: "Database cluster state" — must be one of the valid states

-- Check pg_control file:
-- ls -la $PGDATA/global/pg_control
-- file $PGDATA/global/pg_control     ← should say "data"
-- hexdump -C $PGDATA/global/pg_control | head   ← check for binary content

-- Check filesystem errors:
-- dmesg | grep -i "error\|EIO\|I/O error" | tail -20

-- Check if we have a backup of pg_control:
-- pg_basebackup creates global/pg_control in the backup directory
-- Some backup tools (pgbackrest) also archive pg_control separately

-- Try to use pg_resetwal as last resort (destroys transaction safety):
-- pg_resetwal --dry-run $PGDATA
-- WARNING: This is data-loss territory

Pitfalls

• pg_resetwal (formerly pg_resetxlog) can create a new pg_control with synthetic values, allowing PostgreSQL to start — but the data may be inconsistent and untrustworthy. It is a last resort only.
• After pg_resetwal, PostgreSQL may start but crash recovery cannot be guaranteed. VACUUM, REINDEX, and a dump/restore cycle are required.
• pg_control is updated at every checkpoint. The most recent valid backup copy of pg_control is from the last successful backup — it will be behind the crash state.
• If the storage system itself is failing: fix the hardware/filesystem first, then deal with PostgreSQL. Don’t waste time on recovery if the disk is still failing.
• Data checksums can catch corrupted pg_control on read — but cannot repair it.

Resolution Approach

First: restore pg_control from the most recent backup. Second: if no backup, use pg_resetwal --dry-run to assess the damage, then pg_resetwal as a last resort. After pg_resetwal, treat all data as potentially inconsistent — dump and reload is strongly recommended.