smengcl commented on PR #11246: URL: https://github.com/apache/ozone/pull/11246#issuecomment-5708983594
Thanks @devmadhuu for taking a look. > Does RATIS currently cleanly handles exception errors separately for each raft group. Ratis 3.2.1 does not provide a per-group skip/report startup contract. It starts the groups in parallel and waits for their combined completion before starting the RPC servers. If a group’s startup fails, the overall server startup fails, even if other groups initialized successfully. > We currently run with `raft.server.log.corruption.policy = EXCEPTION` (not overridden), so a corrupt log throws during server.start(). Correct for the default `EXCEPTION` policy. Ratis also supports `WARN_AND_RETURN`, which logs covered segment-read failures and returns the readable entries before the corruption. If Ratis then completes startup successfully, this PR allows the DN to continue normally. The PR preserves both policies: it does not override the configured corruption policy or treat warnings as fatal. It triggers shutdown when startup actually throws. `WARN_AND_RETURN` can still encounter other failures, such as consistency-check failures, and it does not provide per-group isolation. > Would a per‑group skip/report path be safer than failing the entire node? It could improve availability for corruption isolated to one group. However, Ratis would need to prevent the failed group from serving requests, clean up its partially initialized resources, and report the failure so Ozone/SCM can handle the affected pipeline. Shared volume or server failures would also need to be distinguished from isolated group failures. Agreed that this would be a useful separate enhancement. This PR handles the existing server-start failure by terminating the DN with the original cause logged, rather than leaving initialization stuck. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
