Hi, I'm wonder if this isn't going in the wrong direction. We're using CRCs for something they're not well suited for in my understanding - and are paying a reasonably high price for it, given that even hardware accelerated CRCs aren't blazingly fast.
CRCs are used for things like ethernet, iSCSI because they are good at detecting the kinds of errors encountered, namely short bursts of bitflips. And the covered data is limited to a fairly small limit. Which imo makes CRCs a bad choice for WAL. For one, we don't actually expect a short burst of bitflips, the most likely case is all bits after some point changing (because only one part of the record made it to disk). For another, WAL records are *not* limited to a small size, and if anything error detection becomes more important with longer records (they're likely to be span more pages / segments). It's hard to understand, but a nonetheless helpful page is https://users.ece.cmu.edu/~koopman/crc/crc32.html which lists properties for crc32c: https://users.ece.cmu.edu/~koopman/crc/c32/0x8f6e37a0_len.txt which lists (0x8f6e37a0; 0x11edc6f41) <=> (0x82f63b78; 0x105ec76f1) {2147483615,2147483615,5243,5243,177,177,47,47,20,20,8,8,6,6,1,1} | gold | (*op) iSCSI; CRC-32C; CRC-32/4 This cryptic notion AFAIU indicates that for our polynomial we can detect 2bit errors up to a length of 2147483615 bytes, 3 bit errors up to 2147483615, 3 and 4 bit errors up to 5243, 5 and 6 bit errors up to 177, 7/8 bit errors up to 47. IMO for our purposes just about all errors are going to be at least at sector boundaries, i.e. 512 bytes and thus are at least 8 bit large. At that point we are only guaranteed to find a single-byte error (it'll be common to have much more) up to a lenght of 47bits. Which isn't a useful guarantee. With that I perhaps have established that CRC guarantees aren't useful for us. But not yet why we should use something else: Given that we already aren't relying on hard guarantees, we could instead just use a fast hash like xxh3. https://github.com/Cyan4973/xxHash which is fast both for large and small amounts of data. Greetings, Andres Freund
