On Wed, 25 Oct 2023 at 12:20, AVI GROSS via Python-list <python-list@python.org> wrote: > Consider an example of bit rot. I mean what if your CPU or hard disk has a > location where you can write a byte and read it back multiple times and > sometimes get the wrong result. To be really cautions, you might need your > software to write something in multiple locations and when it reads it back > in, check all of them and if most agree, ignore the one or two that don't > while blocking that memory area off and moving your data elsewhere. Or > consider a memory leak that happens rarely but if a program runs for years or > decades, may end up causing an unanticipated error. >
True, but there are FAR more efficient ways to do error correction :) Hamming codes give you single-bit correction and two-bit detection at a cost of log N bits, which is incredibly cheap - even if you were to go for a granularity of 64 bytes (one cache line in a modern Intel CPU), you would need just 11 bits of Hamming code for every 512 bits of data and you can guarantee to fix any single-bit error in any cache line. The "if most agree, ignore the one or two that don't" design implies that you're writing to an absolute minimum of three places, and in order to be able to ignore two that disagree, you'd probably need five copies of everything - that is to say, to store 512 bits of data, you would need 2560 bits of storage. But with a Hamming code, you need just 523 bits to store 512 reliably. Here's a great run-down on how efficiently this can be done, and how easily. https://www.youtube.com/watch?v=X8jsijhllIA Side note: If we assume that random bit flips occur at a rate of one every X storage bits, having redundant copies of data will increase the chances of a failure happening. For example, using a naive and horrendously wasteful "store 256 copies of everything" strategy, you would be 256 times more likely to have a random bitflip, which is insane :) You would also be able to guarantee detection of up to 128 random bitflips. But as you can see, this puts a maximum on your storage ratio. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
