errose28 commented on PR #11861: URL: https://github.com/apache/ozone/pull/11861#issuecomment-6537315791
Thanks for working on this @sarvekshayr. It looks like unregistered DNs being used for reads and writes is an existing issue that ZDU just happened to surface. That is the real root cause we should fix. This is my understanding of the issue: Once the leader SCM has been out of safe mode for long enough, it assumes the unregistered nodes will not arrive and [closes the pipelines periodically](https://github.com/apache/ozone/blob/4bbc178a4ed26d6b4532059b1036c82d662534dd/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManagerImpl.java#L647). However during the 5 minute time window of `HDDS_SCM_WAIT_TIME_AFTER_SAFE_MODE_EXIT`, any nodes that have not yet registered can still have their pipelines handed out for reads and writes, as long as the container in question is open. To fix this on the write path, we should probably filter out of the write candidate pipelines whose nodes have not all registered, probably in the `WritableContainerProvider` implementations. On the read path, we should not return replicas that have not registered. One way to do this could be to create an ad-hoc pipeline with the registered nodes using `createPipelineForRead` even if the container is open, just like we would if the container was closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
