siddhantsangwan commented on code in PR #8460: URL: https://github.com/apache/ozone/pull/8460#discussion_r2099851400
########## hadoop-hdds/docs/content/design/full-volume-handling.md: ########## @@ -0,0 +1,65 @@ +--- +title: Full Volume Handling +summary: Immediately trigger Datanode heartbeat on detecting full volume +date: 2025-05-12 +jira: HDDS-12929 +status: Design +author: Siddhant Sangwan, Sumit Agrawal +--- + +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +## Summary +On detecting a full Datanode volume during write, immediately trigger a heartbeat containing the latest storage report. + +## Problem +When a Datanode volume is close to full, the SCM may not be immediately aware because storage reports are only sent +to it every thirty seconds. This can lead to the SCM allocating multiple blocks to containers on a full DN volume, +causing performance issues when the write fails. The proposal will partly solve this problem. + +In the future (https://issues.apache.org/jira/browse/HDDS-12151) we plan to fail a write if it's going to exceed the min free space boundary in a volume. To prevent this from happening often, SCM needs to stop allocating blocks to containers on such volumes in the first place. + +## Non Goals +The proposed solution describes the complete solution at a high level, however HDDS-12929 will only add the initial Datanode side code for triggering a heartbeat on detecting a full volume + throttling logic. + +Failing the write if it exceeds the min free space boundary is not discussed here. + +## Proposed Solution + +### What does the Datanode do currently? + +In HddsDispatcher, on detecting that the volume being written to is close to full, we add a CloseContainerAction for +that container. This is sent to the SCM in the next heartbeat and makes the SCM close that container. This reaction time + is OK for a container that is close to full, but not if the volume is close to full. + +### Proposal +This is the proposal, explained via a diagram. Review Comment: I had thought of the throttling implementation and even tried it out in code while thinking of the design, but I didn't specify it in the doc. I've added it to the design now. There's also a pull request which implements a part of this design, including throttling - https://github.com/apache/ozone/pull/8492. Please bear with me as I try to balance too much vs too less content in my designs! > When talking about proposed proto updates writing out code examples is helpful. Still thinking about what information we can send over the wire. I'll add it once I've decided. There are a couple of high level ideas to prevent over allocating blocks to a container/volume in the SCM - 1. Track how much allocation is done at the SCM, similar to used space and committed space at the DN. Proposed by @sumitagrawl. 2. Send reports of **open** containers every 30 seconds to SCM, which @ChenSammi proposed. Also handle these reports so that any containers with size >= max size are closed. 3. Decide which containers should be closed in the DN, and send CloseContainerAction for all these containers in the heartbeat (which we briefly discussed in a previous comment). We may need to do some of these or a mix of all of these. As I think it through I'll add more info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For additional commands, e-mail: issues-h...@ozone.apache.org
