Stephen O'Donnell created HDFS-15621:
----------------------------------------
Summary: Datanode DirectoryScanner uses excessive memory
Key: HDFS-15621
URL: https://issues.apache.org/jira/browse/HDFS-15621
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.4.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
Attachments: Screenshot 2020-10-09 at 14.11.36.png
We generally work a rule of 1GB heap on a datanode per 1M blocks. For nodes
with a lot of blocks, this can mean a lot of heap.
We recently captured a heapdump of a DN with about 22M blocks and found only
about 1.5GB was occupied by the ReplicaMap. Another 9GB of the heap is taken by
the DirectoryScanner ScanInfo objects. Most of this memory was alloated to
strings.
Checking the strings in question, we can see two strings per scanInfo, looking
like:
{code}
/current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17/blk_1180438785
_106716708.meta
{code}
I will update a screen shot from MAT showing this.
For the first string especially, the part
"/current/BP-671271071-10.163.205.13-1552020401842/current/finalized/" will be
the same for every block in the block pool as the scanner is only concerned
about finalized blocks.
We can probably also store just the subdir indexes "28" and "27" rather than
"subdir28/subdir17" and then construct the path when it is requested via the
getter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
