errose28 commented on code in PR #8460: URL: https://github.com/apache/ozone/pull/8460#discussion_r2093310170
########## hadoop-hdds/docs/content/design/full-volume-handling.md: ########## @@ -0,0 +1,65 @@ +--- +title: Full Volume Handling +summary: Immediately trigger Datanode heartbeat on detecting full volume +date: 2025-05-12 +jira: HDDS-12929 +status: Design +author: Siddhant Sangwan, Sumit Agrawal +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +## Summary +On detecting a full Datanode volume during write, immediately trigger a heartbeat containing the latest storage report. + +## Problem +When a Datanode volume is close to full, the SCM may not be immediately aware because storage reports are only sent +to it every thirty seconds. This can lead to the SCM allocating multiple blocks to containers on a full DN volume, +causing performance issues when the write fails. The proposal will partly solve this problem. + +In the future (https://issues.apache.org/jira/browse/HDDS-12151) we plan to fail a write if it's going to exceed the min free space boundary in a volume. To prevent this from happening often, SCM needs to stop allocating blocks to containers on such volumes in the first place. + +## Non Goals +The proposed solution describes the complete solution at a high level, however HDDS-12929 will only add the initial Datanode side code for triggering a heartbeat on detecting a full volume + throttling logic. + +Failing the write if it exceeds the min free space boundary is not discussed here. + +## Proposed Solution + +### What does the Datanode do currently? + +In HddsDispatcher, on detecting that the volume being written to is close to full, we add a CloseContainerAction for +that container. This is sent to the SCM in the next heartbeat and makes the SCM close that container. This reaction time + is OK for a container that is close to full, but not if the volume is close to full. + +### Proposal +This is the proposal, explained via a diagram. Review Comment: I don't think the diagram adequately explains the proposal. When talking about proposed proto updates writing out code examples is helpful. Throttling implementation needs to be specified. ########## hadoop-hdds/docs/content/design/full-volume-handling.md: ########## @@ -0,0 +1,65 @@ +--- +title: Full Volume Handling +summary: Immediately trigger Datanode heartbeat on detecting full volume +date: 2025-05-12 +jira: HDDS-12929 +status: Design +author: Siddhant Sangwan, Sumit Agrawal +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +## Summary +On detecting a full Datanode volume during write, immediately trigger a heartbeat containing the latest storage report. + +## Problem +When a Datanode volume is close to full, the SCM may not be immediately aware because storage reports are only sent +to it every thirty seconds. This can lead to the SCM allocating multiple blocks to containers on a full DN volume, +causing performance issues when the write fails. The proposal will partly solve this problem. + +In the future (https://issues.apache.org/jira/browse/HDDS-12151) we plan to fail a write if it's going to exceed the min free space boundary in a volume. To prevent this from happening often, SCM needs to stop allocating blocks to containers on such volumes in the first place. + +## Non Goals +The proposed solution describes the complete solution at a high level, however HDDS-12929 will only add the initial Datanode side code for triggering a heartbeat on detecting a full volume + throttling logic. + +Failing the write if it exceeds the min free space boundary is not discussed here. + +## Proposed Solution + +### What does the Datanode do currently? + +In HddsDispatcher, on detecting that the volume being written to is close to full, we add a CloseContainerAction for +that container. This is sent to the SCM in the next heartbeat and makes the SCM close that container. This reaction time + is OK for a container that is close to full, but not if the volume is close to full. Review Comment: We also have the ability to immediately [trigger a heartbeat at any time](https://github.com/apache/ozone/blob/2b48e8c6ec1739d541d5c02183ad1a91d9f7a308/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java#L566). We do this for [volume failure](https://github.com/apache/ozone/blob/a2ad1e334dd5a887cc7a451671f38bfdf91c6aa0/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java#L314) already. Seems the issue could be resolved by leaving the `CloseContainerAction` handling as is and just calling this method when the volume is getting full. This leaves the volume to container mapping inside the datanode without needing to add it to SCM. ########## hadoop-hdds/docs/content/design/full-volume-handling.md: ########## @@ -0,0 +1,65 @@ +--- +title: Full Volume Handling +summary: Immediately trigger Datanode heartbeat on detecting full volume +date: 2025-05-12 +jira: HDDS-12929 +status: Design +author: Siddhant Sangwan, Sumit Agrawal +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +## Summary +On detecting a full Datanode volume during write, immediately trigger a heartbeat containing the latest storage report. + +## Problem +When a Datanode volume is close to full, the SCM may not be immediately aware because storage reports are only sent +to it every thirty seconds. This can lead to the SCM allocating multiple blocks to containers on a full DN volume, +causing performance issues when the write fails. The proposal will partly solve this problem. + +In the future (https://issues.apache.org/jira/browse/HDDS-12151) we plan to fail a write if it's going to exceed the min free space boundary in a volume. To prevent this from happening often, SCM needs to stop allocating blocks to containers on such volumes in the first place. + +## Non Goals +The proposed solution describes the complete solution at a high level, however HDDS-12929 will only add the initial Datanode side code for triggering a heartbeat on detecting a full volume + throttling logic. + +Failing the write if it exceeds the min free space boundary is not discussed here. + +## Proposed Solution + +### What does the Datanode do currently? + +In HddsDispatcher, on detecting that the volume being written to is close to full, we add a CloseContainerAction for +that container. This is sent to the SCM in the next heartbeat and makes the SCM close that container. This reaction time + is OK for a container that is close to full, but not if the volume is close to full. Review Comment: How does this account for the 5GB of reserved/committed space for open containers? I believe that will be counted against the volume's capacity as well. We need to break down what measures are being used to consider a "full" volume. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For additional commands, e-mail: issues-h...@ozone.apache.org
