This is a QA system and unfortunately there is no recent backup.... So as a last resort I am looking for any way to bring up Postgres when it has corrupt data in it:
FATAL: could not remove old lock file "postmaster.pid": Read-only file system HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again. Jun 8 06:43:16 mxlqa401 postgres[21401]: [1-1] FATAL: could not remove old lock file "postmaster.pid": Read-only file system Jun 8 06:43:16 mxlqa401 postgres[21401]: [1-2] HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again . FATAL: could not remove old lock file "postmaster.pid": Read-only file system HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again. Jun 8 06:43:29 mxlqa401 postgres[21476]: [1-1] FATAL: could not remove old lock file "postmaster.pid": Read-only file system Jun 8 06:43:29 mxlqa401 postgres[21476]: [1-2] HINT: The file seems accidentally left over, but it could not be removed. Please remove the file by hand and try again . Jun 8 06:44:23 mxlqa401 postgres[21520]: [1-1] LOG: database system was interrupted at 2009-06-05 21:52:54 MDT Jun 8 06:44:24 mxlqa401 postgres[21520]: [2-1] LOG: checkpoint record is at 134/682530F0 Jun 8 06:44:24 mxlqa401 postgres[21520]: [3-1] LOG: redo record is at 134/682530F0; undo record is at 0/0; shutdown FALSE Jun 8 06:44:24 mxlqa401 postgres[21520]: [4-1] LOG: next transaction ID: 3005778382; next OID: 103111004 Jun 8 06:44:24 mxlqa401 postgres[21520]: [5-1] LOG: next MultiXactId: 93647; next MultiXactOffset: 190825 Jun 8 06:44:24 mxlqa401 postgres[21520]: [6-1] LOG: database system was not properly shut down; automatic recovery in progress Jun 8 06:44:24 mxlqa401 postgres[21520]: [7-1] LOG: redo starts at 134/68253134 Jun 8 06:44:24 mxlqa401 postgres[21520]: [8-1] PANIC: could not access status of transaction 3005778383 Jun 8 06:44:24 mxlqa401 postgres[21520]: [8-2] DETAIL: could not read from file "pg_clog/0B32" at offset 139264: Success Jun 8 06:44:29 mxlqa401 postgres[21518]: [1-1] LOG: startup process (PID 21520) was terminated by signal 6 Jun 8 06:44:29 mxlqa401 postgres[21518]: [2-1] LOG: aborting startup due to startup process failure Jun 8 06:44:36 mxlqa401 postgres[21574]: [1-1] LOG: database system was interrupted while in recovery at 2009-06-08 06:44:24 MDT Jun 8 06:44:36 mxlqa401 postgres[21574]: [1-2] HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery. Jun 8 06:44:36 mxlqa401 postgres[21574]: [2-1] LOG: checkpoint record is at 134/682530F0 Jun 8 06:44:36 mxlqa401 postgres[21574]: [3-1] LOG: redo record is at 134/682530F0; undo record is at 0/0; shutdown FALSE Jun 8 06:44:36 mxlqa401 postgres[21574]: [4-1] LOG: next transaction ID: 3005778382; next OID: 103111004 Jun 8 06:44:36 mxlqa401 postgres[21574]: [5-1] LOG: next MultiXactId: 93647; next MultiXactOffset: 190825 Jun 8 06:44:36 mxlqa401 postgres[21574]: [6-1] LOG: database system was not properly shut down; automatic recovery in progress I tried to bring up a postgres backend process to get into the database in single-user mode and that won't work either: bash-3.2$ postgres -D /mxl/var/pgsql/data PANIC: could not access status of transaction 3005778382 DETAIL: could not read from file "pg_clog/0B32" at offset 139264: Success Aborted bash-3.2$ postgres -D /mxl/var/pgsql/data -d 5 postgres PANIC: could not access status of transaction 3005778382 DETAIL: could not read from file "pg_clog/0B32" at offset 139264: Success Aborted Any suggestions other than the obvious (restore from backup) would be appreciated. Thanks, Keaton
