Yesterday morning, one of my streaming replication slaves running 9.2.3 crashed with the following in the log file:
2013-03-28 12:49:30 GMT WARNING: page 1441792 of relation base/63229/63370 does not exist 2013-03-28 12:49:30 GMT CONTEXT: xlog redo delete: index 1663/63229/109956; iblk 303, heap 1663/63229/63370; 2013-03-28 12:49:30 GMT PANIC: WAL contains references to invalid pages 2013-03-28 12:49:30 GMT CONTEXT: xlog redo delete: index 1663/63229/109956; iblk 303, heap 1663/63229/63370; 2013-03-28 12:49:31 GMT LOG: startup process (PID 22941) was terminated by signal 6: Aborted 2013-03-28 12:49:31 GMT LOG: terminating any other active server processes 2013-03-28 12:49:31 GMT WARNING: terminating connection because of crash of another server process 2013-03-28 12:49:31 GMT DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2013-03-28 12:49:31 GMT HINT: In a moment you should be able to reconnect to the database and repeat your command. 2013-03-28 12:57:44 GMT LOG: database system was interrupted while in recovery at log time 2013-03-28 12:37:42 GMT 2013-03-28 12:57:44 GMT HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target. 2013-03-28 12:57:44 GMT LOG: entering standby mode 2013-03-28 12:57:44 GMT LOG: redo starts at 19/2367CE30 2013-03-28 12:57:44 GMT LOG: incomplete startup packet 2013-03-28 12:57:44 GMT LOG: consistent recovery state reached at 19/241835B0 2013-03-28 12:57:44 GMT LOG: database system is ready to accept read only connections 2013-03-28 12:57:44 GMT LOG: invalid record length at 19/2419EE38 2013-03-28 12:57:44 GMT LOG: streaming replication successfully connected to primary As you can see I was able to restart it and it picked up and synchronized right away, but this crash still concerns me. The DB has about 75GB of data in it, and it is almost entirely write traffic. It's essentially a log aggregator. I believe it was doing a pg_dump backup at the time of the crash. It has hot_standby_feedback on to allow that process to complete. Any insights into this, or advice on figuring out the root of it would be appreciated. So far all the things I've found like this are bugs that should be fixed in this version, or the internet equivalent of a shrug. Thanks! QH
