[
https://issues.apache.org/jira/browse/HDDS-15330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-15330:
----------------------------------
Labels: pull-request-available (was: )
> Implement SCM FCR rate limit
> ----------------------------
>
> Key: HDDS-15330
> URL: https://issues.apache.org/jira/browse/HDDS-15330
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Labels: pull-request-available
> Attachments: scm_Leak_Suspects_oom.zip, scm_System_Overview_oom.zip,
> scm_Top_Components_oom.zip
>
>
> We have previous instances where a new bootstrapped SCM encountered OOM (FYI
> the OOM has 96GB heap size).
> See the attached MAT tool ParseHeapDump.sh output. The possible issue is that
> FCRs are currently stored in
> 1. ContainerReportQueue (at most 1, this accounts to 15.2 GB)
> 2. Hadoop RPC handlers / Server$Handler(ozone.scm.datanode.handler.count.key
> default to 100 and this accounts to 39.7 GB)
> 3. Hadoop RPC server call queue / ProtobufRpcEngine$Server (this accounts for
> 9.4 GB)
> 4. Hadoop RPC connection / Server$Connection (this accounts for 6.0 GB)
> If a lot of DN sends FCR at the same time (which should happen for newly
> restarted / bootstrapped SCM), this will trigger bursty FCR reports from all
> DNs, which can trigger memory issue.
> HDFS implements a full block reports rate limit in HDFS-7923 to reduce the
> concurrent block reports residing in SCM using BlockReportLeaseManager. Ozone
> should also implement similar mechanism to prevent FCR storms. A possible
> design is that we register DN first, but don't include the full FCR
> immediately. SCM grants only N datanodes permission to send FCRs at once,
> similar to HDFS implementation. This way, we can ensure that N FCR can exists
> at one time in SCM heap memory.
> Another possibility to reduce the single FCR size to to split the FCR to one
> FCR per volume (can be considered in the future).
> One tradeoff of the rate-limiting is that new SCM might delay the SafeMode
> exit. However, this is better than SCM OOM. Another tradeoff is that FCR
> might be delayed for large cluster (we need to think about this). Another
> issue mentioned in HDDS-7923 is that there might be some DNs that might take
> a long time to send its FCR to the SCM since it never gets the lease, but
> seems HDFS has not handled it as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
