Chu Cheng Li created HDDS-16634:
-----------------------------------
Summary: Decommission can take several times longer when only a
few datanodes have free space
Key: HDDS-16634
URL: https://issues.apache.org/jira/browse/HDDS-16634
Project: Apache Ozone
Issue Type: Improvement
Components: SCM
Reporter: Chu Cheng Li
When only a small share of datanodes can take a new replica, for example in a
nearly full cluster where a few new datanodes were added,
SCMContainerPlacementRackAware often fails to find a target although suitable
datanodes exist. It picks datanodes at random and gives up after 3 unsuitable
picks at each step (the other racks, then any rack). Full, decommissioning and
in-maintenance datanodes stay in the network topology, so most picks land on
them. ReplicationManager then retries the container on its next run, 30 seconds
later by default, and the same can happen again. Decommission and other repairs
still finish, but they can take several times longer.
This was found with the SCM simulation (HDDS-16627), with HDDS-16623,
HDDS-16628 and HDDS-16629 applied. Setup: 1 GB containers, ReplicationManager
intervals at their defaults (30 s for under-replicated containers, 5 min for
the full run), no faults. After an hour of writes, every datanode except a few
reports no free space, and 5 minutes later the busiest full datanodes (23 to 28
containers each) are decommissioned. The comparison is the same code where
unsuitable picks do not use up a retry, so a suitable datanode is always found
when one exists. Times are simulated, 5 seeds each:
||Datanodes||With free space||Decommissioned||Time to finish||Same, with no
misses||Slower by (median)||
|240 (10 racks)|2 (0.8%)|1|790 - 3400 s|450 - 810 s|4.4x|
|240 (10 racks)|8 (3.3%)|1|630 - 1000 s|450 - 810 s|1.4x|
|240 (10 racks)|24 (10%)|1|510 - 840 s|460 - 810 s|1.0x|
|240 (10 racks)|8 (3.3%)|4 at once|1020 - 1890 s|460 - 780 s|2.3x|
|120 (6 racks)|4 (3.3%)|1|550 - 1030 s|420 - 510 s|1.3x|
|60 (4 racks)|2 (3.3%)|1|610 - 1210 s|480 - 540 s|1.3x|
Placement failed 17 to 800 times per run while suitable datanodes existed. With
10% of the datanodes free, the effect is gone.
Not counting unsuitable picks as retries is not a good fix on its own:
NetworkTopologyImpl.chooseRandom gets slower with every excluded node, so
searching a large, mostly full cluster that way is expensive (about 7 s per
placement was measured at 1000 datanodes when almost none could take a
replica). A fix probably needs a cheaper way to choose only among suitable
datanodes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
