On Fri, Jul 4, 2025 at 9:49 AM Erik Johnston <er...@element.io> wrote:
> Hi, a quick update: > > - We have discovered that the corruption was present from before libicu > update. > - We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed > matrix`, which returned nothing > - We believe that means that (and matches what we see sampling) the index > has gained extra entries, i.e. that for a given state group it does return > all the relevant rows in the table *plus* extra rows. > > We are also seeing old state groups starting to point at rows that have > only just been inserted. For example, querying for 353864583 on the primary > it returns that row plus four rows that have been inserted today, but on > the backup from last week an index only scan for 353864583 only returns one > row. This makes it feel like the corruption is ongoing? Nothing should have > modified that state group in the interim (they are generally immutable). > > This naively feels like when inserting a new row we sometimes add the row > to the index twice: once pointing from the correct state group to the new > row, and once from an old state group to the new row? > > Are checksums enabled in the instance? -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster!
