[
https://issues.apache.org/jira/browse/HDDS-16634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahesh kumar gaddam reassigned HDDS-16634:
------------------------------------------
Assignee: Mahesh kumar gaddam
> Decommission can take several times longer when only a few datanodes have
> free space
> ------------------------------------------------------------------------------------
>
> Key: HDDS-16634
> URL: https://issues.apache.org/jira/browse/HDDS-16634
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Mahesh kumar gaddam
> Priority: Major
>
> When only a small share of datanodes can take a new replica, for example in a
> nearly full cluster where a few new datanodes were added,
> SCMContainerPlacementRackAware often fails to find a target although suitable
> datanodes exist. It picks datanodes at random and gives up after 3 unsuitable
> picks at each step (the other racks, then any rack). Full, decommissioning
> and in-maintenance datanodes stay in the network topology, so most picks land
> on them. ReplicationManager then retries the container on its next run, 30
> seconds later by default, and the same can happen again. Decommission and
> other repairs still finish, but they can take several times longer.
> This was found with the SCM simulation (HDDS-16627), with HDDS-16623,
> HDDS-16628 and HDDS-16629 applied. Setup: 1 GB containers, ReplicationManager
> intervals at their defaults (30 s for under-replicated containers, 5 min for
> the full run), no faults. After an hour of writes, every datanode except a
> few reports no free space, and 5 minutes later the busiest full datanodes (23
> to 28 containers each) are decommissioned. The comparison is the same code
> where unsuitable picks do not use up a retry, so a suitable datanode is
> always found when one exists. Times are simulated, 5 seeds each:
> ||Datanodes||With free space||Decommissioned||Time to finish||Same, with no
> misses||Slower by (median)||
> |240 (10 racks)|2 (0.8%)|1|790 - 3400 s|450 - 810 s|4.4x|
> |240 (10 racks)|8 (3.3%)|1|630 - 1000 s|450 - 810 s|1.4x|
> |240 (10 racks)|24 (10%)|1|510 - 840 s|460 - 810 s|1.0x|
> |240 (10 racks)|8 (3.3%)|4 at once|1020 - 1890 s|460 - 780 s|2.3x|
> |120 (6 racks)|4 (3.3%)|1|550 - 1030 s|420 - 510 s|1.3x|
> |60 (4 racks)|2 (3.3%)|1|610 - 1210 s|480 - 540 s|1.3x|
> Placement failed 17 to 800 times per run while suitable datanodes existed.
> With 10% of the datanodes free, the effect is gone.
> Not counting unsuitable picks as retries is not a good fix on its own:
> NetworkTopologyImpl.chooseRandom gets slower with every excluded node, so
> searching a large, mostly full cluster that way is expensive (about 7 s per
> placement was measured at 1000 datanodes when almost none could take a
> replica). A fix probably needs a cheaper way to choose only among suitable
> datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
