On Thu, 2023-04-06 at 16:41 +0000, Evgeny Morozov wrote: > Our PostgreSQL 15.2 instance running on Ubuntu 18.04 has crashed with this > error: > > 2023-04-05 09:24:03.448 UTC [15227] ERROR: index "pg_class_oid_index" > contains unexpected zero page at block 0 > [...] > > We had the same thing happened about a month ago on a different database on > the same cluster. > For a while PG actually ran OK as long as you didn't access that specific DB, > but when trying > to back up that DB with pg_dump it would crash every time. At that time one > of the disks > hosting the ZFS dataset with the PG data directory on it was reporting > errors, so we thought > it was likely due to that. > > Unfortunately, before we could replace the disks, PG crashed completely and > would not start > again at all, so I had to rebuild the cluster from scratch and restore from > pg_dump backups > (still onto the old, bad disks). Once the disks were replaced (all of them) I > just copied > the data to them using zfs send | zfs receive and didn't bother restoring > pg_dump backups > again - which was perhaps foolish in hindsight. > > Well, yesterday it happened again. The server still restarted OK, so I took > fresh pg_dump > backups of the databases we care about (which ran fine), rebuilt the cluster > and restored > the pg_dump backups again - now onto the new disks, which are not reporting > any problems. > > So while everything is up and running now this error has me rather concerned. > Could the > error we're seeing now have been caused by some corruption in the PG data > that's been there > for a month (so it could still be attributed to the bad disk), which should > now be fixed by > having restored from backups onto good disks?
Yes, that is entirely possible. > Could this be a PG bug? It could be, but data corruption caused by bad hardware is much more likely. > What can I do to figure out why this is happening and prevent it from > happening again? No idea about the former, but bad hardware is a good enough explanation. As to keeping it from happening: use good hardware. Yours, Laurenz Albe
