Ray Mattingly created HBASE-30299:
-------------------------------------
Summary: SimpleRegionNormalizer should not attempt to merge
primary and secondary replicas
Key: HBASE-30299
URL: https://issues.apache.org/jira/browse/HBASE-30299
Project: HBase
Issue Type: Improvement
Affects Versions: 2.6.6
Reporter: Ray Mattingly
Assignee: Ray Mattingly
Right now the SimpleRegionNormalizer will naively try to merge primary and
secondary replicas, which is obviously not valid. These invalid normalization
steps are rejected, but it's a waste of time, obviously errant, and can prevent
normalization from ever successfully completing when secondary replicas are
enabled.
We can and should trivially fix this by making the SimpleRegionNormalizer
capable of differentiating between primary and secondary replicas. I believe we
can do this succinctly enough for it to still qualify as "simple" 😁
For reference, here is the corresponding stack trace upon rejection:
{code:java}
2026-07-22T05:46:57,165 [normalizer-worker-0] INFO
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorker: failed to
submit plan
MergeNormalizationPlan[normalizationTargets=[NormalizationTarget[regionInfo={ENCODED
=> 99e14a2c6e7cd493a488bf99bd79fd2b, NAME =>
'associations-3,<REDACTED>,1755987304406.99e14a2c6e7cd493a488bf99bd79fd2b.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=4170],
NormalizationTarget[regionInfo={ENCODED => ed03ce7a152654b546114f2c7f753f38,
NAME =>
'associations-3,<REDACTED>,1755987304406_0001.ed03ce7a152654b546114f2c7f753f38.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID =>
1},regionSizeMb=4170], NormalizationTarget[regionInfo={ENCODED =>
28d5389abcd870a47fc70b6ec8b1eabb, NAME =>
'associations-3,<REDACTED>,1755986877808.28d5389abcd870a47fc70b6ec8b1eabb.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=4196],
NormalizationTarget[regionInfo={ENCODED => 4ed85d4429e757789ac8bf06cf348e1d,
NAME =>
'associations-3,<REDACTED>,1755986877808_0001.4ed85d4429e757789ac8bf06cf348e1d.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID =>
1},regionSizeMb=4196], NormalizationTarget[regionInfo={ENCODED =>
664aeafa73baa80dc079fb3b8d43ea4a, NAME =>
'associations-3,<REDACTED>,1766134153860.664aeafa73baa80dc079fb3b8d43ea4a.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=2081],
NormalizationTarget[regionInfo={ENCODED => 7dc8f3f9027c362b7097fbc14a8bad76,
NAME =>
'associations-3,<REDACTED>,1766134153860_0001.7dc8f3f9027c362b7097fbc14a8bad76.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID =>
1},regionSizeMb=2081], NormalizationTarget[regionInfo={ENCODED =>
2c0b0bf4302b61ed3fc170be7aae0592, NAME =>
'associations-3,<REDACTED>,1766134153860.2c0b0bf4302b61ed3fc170be7aae0592.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=2083],
NormalizationTarget[regionInfo={ENCODED => 0577993d4ff3ca839adb03a1f8d11951,
NAME =>
'associations-3,<REDACTED>,1766134153860_0001.0577993d4ff3ca839adb03a1f8d11951.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID =>
1},regionSizeMb=2083], NormalizationTarget[regionInfo={ENCODED =>
880a224d76719aabb6d8c18b8e74567c, NAME =>
'associations-3,<REDACTED>,1754972586167.880a224d76719aabb6d8c18b8e74567c.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=4125],
NormalizationTarget[regionInfo={ENCODED => ded028dca4cc4a802d89f8c69ef0d581,
NAME =>
'associations-3,<REDACTED>,1754972586167_0001.ded028dca4cc4a802d89f8c69ef0d581.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID =>
1},regionSizeMb=4125], NormalizationTarget[regionInfo={ENCODED =>
75b73186c0878f9cc7f01de49c7497ef, NAME =>
'associations-3,<REDACTED>,1761802096163.75b73186c0878f9cc7f01de49c7497ef.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>'},regionSizeMb=4100]]].
org.apache.hadoop.hbase.exceptions.MergeRegionException: Can't merge
non-default replicas; {ENCODED => ed03ce7a152654b546114f2c7f753f38, NAME =>
'associations-3,<REDACTED>,1755987304406_0001.ed03ce7a152654b546114f2c7f753f38.',
STARTKEY => '<REDACTED>', ENDKEY => '<REDACTED>', REPLICA_ID => 1}
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.checkRegionsToMerge(MergeTableRegionsProcedure.java:147)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.<init>(MergeTableRegionsProcedure.java:107)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:2246)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.HMaster.mergeRegions(HMaster.java:2240)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorker.submitMergePlan(RegionNormalizerWorker.java:311)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorker.submitPlans(RegionNormalizerWorker.java:273)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorker.run(RegionNormalizerWorker.java:198)
~[hbase-server-2.6-hubspot-SNAPSHOT.jar:2.6-hubspot-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
