The GitHub Actions job "CI" on kvrocks.git has failed. Run started by GitHub user git-hulk (triggered by git-hulk).
Head commit for run: 5cdbb1954657ba387490adeef2af361b744b21be / Myth <caipen...@outlook.com> Try to recover automatically from SST corruption background error(#1667) When the SST file corrupts, which is an unrecoverable error for the rocksdb, then rocksdb will go into read-only mode(https://github.com/facebook/rocksdb/wiki/Background-Error-Handling). Only restart rocksdb to try to recover from the error. When does sst file corruption occur? The error message looks like this: ``` 1. Corruption: Corrupt or unsupported format_version: 1005 in /tmp/kvrocks/data/db/000038.sst 2. Corruption: Bad table magic number: expected 9863518390377041911, found 9863518390377041912 in /tmp/kvrocks_db/data/db/000038.sst 3. Corruption: block checksum mismatch: stored = 3308200672, computed = 51173877, type = 4 in /tmp/kvrocks_db/data/db/000038.sst offset 0 size 15715 ``` The cause of the error is usually a hardware issue or a problem with the network or cloud disk (when using the cloud disk). The most common place we see this error is when a file is generated by `Compaction` or `Flush` and the `Version` applies the result. In this case, the result of the compaction is not actually applied, so we can ignore the error and avoid restarting the rocksdb. Tikv introduces this check when sst file corruption occurs, you can refer to: - https://github.com/tikv/tikv/issues/10578 - https://github.com/tikv/tikv/pull/10961 Let's try it on Kvrocks: 1. Extract the sst file from the background error message 2. Determine if it is a living file 3. If not, we ignore the error and force recovery from the background error For the rocksdb error message, before the rocksdb v7.10.2, the error message was imperfect and we could only recover from a limited number of errors. Thanks to this PR https://github.com/facebook/rocksdb/pull/11009, the error message is enriched and we can recover from more scenarios. Report URL: https://github.com/apache/kvrocks/actions/runs/5864170050 With regards, GitHub Actions via GitBox
