Late yesterday afternoon our DB server went down hard. we tried to re-start and it went into recovery mode to recover transaction history and failed. Notable error was:
FATAL: failed to re-find parent key in index "257969064" for split pages 8366/12375 If you look this error up, it indicates issues with the transaction logs and the inability to recover due to corrupt or missing transaction logs. The solution is to: 1. Back up the DB files 2. Run pg_resetxlogs (this might produce corruption due inconsistent data) 3. Dump the DB 4. Reload the DB Unfortunately this solution is not practical in our case for multiple reasons. - We do not have the space for steps 1 & 3 - The time required for steps 1, 3, and 4 is approximately 1 week per step (3 weeks total) since our database size is approximately 5.4TB If the data is in an inconsistent state, are there other alternative solutions, such as finding the index specified in the FATAL error and somehow dropping it? and does anyone know what circumstances/conditions might corrupt or cause a transaction log to go missing. thanks
