On Fri, 4 Jul 2025, 15:38 Ron Johnson, <ronljohnso...@gmail.com> wrote:
> On Fri, Jul 4, 2025 at 9:49 AM Erik Johnston <er...@element.io> wrote: > >> Hi, a quick update: >> >> - We have discovered that the corruption was present from before libicu >> update. >> - We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed >> matrix`, which returned nothing >> - We believe that means that (and matches what we see sampling) the index >> has gained extra entries, i.e. that for a given state group it does return >> all the relevant rows in the table *plus* extra rows. >> >> We are also seeing old state groups starting to point at rows that have >> only just been inserted. For example, querying for 353864583 on the primary >> it returns that row plus four rows that have been inserted today, but on >> the backup from last week an index only scan for 353864583 only returns one >> row. This makes it feel like the corruption is ongoing? Nothing should have >> modified that state group in the interim (they are generally immutable). >> >> This naively feels like when inserting a new row we sometimes add the row >> to the index twice: once pointing from the correct state group to the new >> row, and once from an old state group to the new row? >> >> > Are checksums enabled in the instance? > Alas not. We've also now found that the index on the backup does in fact point to those ctids after all, but they are marked as dead. So at some point between then and when we inserted the new row at that ctid today those entries were marked undead. -- Copyright © 2025 Element - All rights reserved. The Element name, logo and device are registered trademarks of New Vector Ltd. Registered number: 10873661. Registered in England and Wales. Registered address: 10 Queen Street Place, London, United Kingdom, EC4R 1AG. This message is intended for the addressee only and may contain private and confidential information or material which may be privileged. If this message has come to you in error please delete it immediately and do not copy it or show it to any other person.
