ivandika3 opened a new pull request, #11905: URL: https://github.com/apache/ozone/pull/11905
## What changes were proposed in this pull request? This patch introduces the FCR lease mechanism similar to HDFS block report lease mechanism to bound the memory overhead of SCM due to full container reports. See https://issues.apache.org/jira/browse/HDDS-15330 for the background. ### Overall flow 1. A datanode marks its FCR pending / deferred (see `fullContainerReportDeferred`) and requests a lease through a normal heartbeat 2. SCM grants a nonzero lease ID and term if eligible and capacity is available 1. `ozone.scm.max.full.container.report.leases` is the maximum number of concurrent FCR that SCM allows at one time. It acts as the rate limit 3. The datanode immediately sends a follow-up heartbeat containing the FCR and the FCR lease. 4. SCM validates the lease, then queues and processes the report 5. Completion or discard release the lease and allow new FCR to be handled. If no lease is granted, the datanode retries on subsequent heartbeats. Other reports and commands continue normally. ### Lease Lifecycle The following is the lease lifecycleYou can refer to the `SCMFullContainerReportLeaseManager` (state is in `LeaseState`). The main flow is as follow ``` text NO LEASE | | Lease request granted: reserve capacity v GRANTED | | Valid leased FCR received: claimLease() v CLAIMED | | Report handler starts: startProcessing() v PROCESSING | | Processing succeeds or fails: complete(processed) v NO LEASE ``` However, there are also other transitions (e.g. lease expiry, replacement request, new registration, etc). ```text GRANTED --[replacement request granted; new lease ID]--> GRANTED GRANTED --[grant expiry detected]----------------------> NO LEASE GRANTED --[lease request prunes grants from other terms]> NO LEASE GRANTED --[new deferred registration]------------------> NO LEASE GRANTED --[datanode removed from SCM]------------------> NO LEASE CLAIMED --[dispatch rejected or publication fails]-----> NO LEASE CLAIMED --[queued report discarded or queue cleared]---> NO LEASE ``` Notable properties - `GRANTED`, `CLAIMED`, and `PROCESSING` all reserve capacity - `CLAIMED` will be invoked before it's queued in the EventQueue to be processed - Only `GRANTED` expires since once it goes to `CLAIMED` and `PROCESSING`, the FCR has been received and should be completely handled by SCM (not waiting for datanode). - Expiry is checked during lease requests or report validation, not by a timer. - This is to simplify the logic. Having a thread to expire the leases would need more bookkeepings. The number of leases should not be too large so we can afford outstanding leases and we can allow lease operation to expire the existing leases. ### SCM HA The FCR lease is not replicated in SCM and the lease ID is not generated using the SCM HA sequence ID generator. This is because container report (FCR and ICR) processing is handled by both the SCM leader and followers. Therefore, we only need that the lease ID for each SCM node to be different. ### Cross Cutting Concerns - Feature flag - `ozone.scm.fullcontainer.report.rate.limit.enabled` flag is introduced with default disabled. This would allow us to revert back to accepting all FCR report if for example it causes SCM to be in SafeMode longer than necessary. - `ozone.scm.fullcontainer.report.rate.limit.enabled` is reconfigurable per SCM node. This allows us to evaluate the FCR lease mechanism and disable it quickly if there are issues. - Compatibility: This patch should allow rolling upgrade - New SCM and old datanode: Old datanode immediately sends FCR. In that case, SCM would bypass the lease mechanism and handle the container report immediately. - Old SCM and new datanode: The datanode would be able to check whether SCM supports the FCR lease feature through the (through `SCM_FCR_LEASE_SUPPORTED` in the `VersionResponse` as well as ### Evaluations There are tradeoffs on enabling the FCR lease mechanism Advantages - The number of FCR that can exist in SCM (RPC queue, Event queue, processing) is bounded by `ozone.scm.max.full.container.report.leases` - Therefore in a cluster with a lot of dense nodes or a lot of small containers, this would increase the memory requirement of SCM (cost saving) and allow more stable SCM (less JVM pressure from container reports) Disadvantages - SafeMode requires that a certain number of containers are going to be recorded. The FCR lease mechanism can increase the time it takes for SCM to exit SafeMode. - We can either reduce the heartbeat interval or increase the FCR lease limit for this, or disable the feature flag if it's too long ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15330 ## How was this patch tested? UT, IT. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
