Jason Lowe created HADOOP-14412:
-----------------------------------
Summary: HostsFileReader#getHostDetails is very expensive on large
clusters
Key: HADOOP-14412
URL: https://issues.apache.org/jira/browse/HADOOP-14412
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 2.8.0
Reporter: Jason Lowe
Assignee: Jason Lowe
After upgrading one of our large clusters to 2.8 we noticed many IPC server
threads of the resourcemanager spending time in NodesListManager#isValidNode
which in turn was calling HostsFileReader#getHostDetails. The latter is
creating complete copies of the include and exclude sets for every node
heartbeat, and these sets are not small due to the size of the cluster. These
copies are causing multiple resizes of the underlying HashSets being filled and
creating lots of garbage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
