farmmamba created HDFS-17484:
--------------------------------
Summary: Introduce redundancy.considerLoad.minLoad to avoiding
excluding nodes when they are not busy actually
Key: HDFS-17484
URL: https://issues.apache.org/jira/browse/HDFS-17484
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.4.0
Reporter: farmmamba
Assignee: farmmamba
Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by
default, and
dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default.
Think about below situation. when we are doing stress test, we may deploy hdfs
client onto the datanode. So, this hdfs client will prefer to write to its
local datanode and increase this machine's load. Suppose we have 3 datanodes,
the load of them are as below: 5.0, 0.2, 0.3.
The load equals to 5.0 will be excluded when choose datanodes for a block. But
actually, it is not slow node when load equals to 5.0 for a machine with 80 cpu
cores.
So, we should better add a new configuration entry :
`dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor
we will make considerLoad take effect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
