[
https://issues.apache.org/jira/browse/HBASE-30323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-30323:
-----------------------------------
Labels: pull-request-available (was: )
> [RSGroup] Forward-port HBASE-22658 to branch-2
> ----------------------------------------------
>
> Key: HBASE-30323
> URL: https://issues.apache.org/jira/browse/HBASE-30323
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.6, 2.2.3, 2.1.9
> Reporter: Umesh Kumar Kumawat
> Assignee: Umesh Kumar Kumawat
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0-alpha-1, 3.0.0
>
>
> region_mover.rb should only choose same rsgroup servers as target servers.
> In current case HMaster need to correct it. I think we can skip the process
> where HMaster tries to figure out the target host if we pass the right one.
>
> As HMaster do have one layer of protection, on its own it doesn't post any
> issue but if this issue comes with other issue like "drainig host is not
> added to draining list", then it can cause unavailibility.
> We faced it in out prod where we use different RS group for system tables ie
> hbase:meta.
> Lets say hbase:meta is on rs-sys-1.
> Before start of draining the operation where we add it to draining list
> failed. Draining script tried to drain the rs-sys-1. it tried to assign the
> hbase:meta on rs-x. HMaster prevented it but it choose rs-sys-0 only that was
> about to be killed.
> This caused hbase:meta to unavailable for some time. And rolling restart at
> the same time can cause more unavailability of other data regions.
>
> draining_servers.rb add command timed out so HMaster was not aware of it.
> (We can have some improvements here as well. Going ahead with drainig when
> HMaster is not aware is also not right)
> {code:java}
> 2026-06-03 14:07:29,341 INFO [PEWorker-39]
> assignment.TransitRegionStateProcedure - Starting pid=193574338,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE;
> state=CLOSED, location=localhost,1,1; forceNewPlan=false, retain=false
>
> 2026-06-03 14:07:28,550 INFO [PEWorker-54] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=193574380, ppid=193574338, state=RUNNABLE;
> CloseRegionProcedure 1588230740, server=regionserver-sys-1}]
>
> 2026-06-03 14:07:29,504 INFO [PEWorker-27] procedure2.ProcedureExecutor -
> Initialized subprocedures=[{pid=193574527, ppid=193574338, state=RUNNABLE;
> OpenRegionProcedure 1588230740, server=regionserver-sys-1}]
> {code}
> {{As this RS was already scheduled to be killed. It got killed after some
> time. }}
> {code:java}
> 2026-06-03 14:10:39,022 INFO [PEWorker-18] procedure.MasterProcedureScheduler
> - Took xlock for pid=193622058, ppid=193622049,
> state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
