[
https://issues.apache.org/jira/browse/HDFS-17818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18015127#comment-18015127
]
ASF GitHub Bot commented on HDFS-17818:
---------------------------------------
lfxy commented on PR #7862:
URL: https://github.com/apache/hadoop/pull/7862#issuecomment-3205673124
@Hexiaoqiao @ayushtkn @tomscut Do you think that uploading fsimage in
checkpoint with observer namenode should be changed from serial to parallel?
In our production cluster, there are one active, one standby, and two
oberver namenodes. The serial uploading takes 40 minutes longer than the
parallel operation.
> Fix serial fsimage transfer during checkpoint with multiple namenodes
> ---------------------------------------------------------------------
>
> Key: HDFS-17818
> URL: https://issues.apache.org/jira/browse/HDFS-17818
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.5.0
> Reporter: caozhiqiang
> Assignee: caozhiqiang
> Priority: Major
> Labels: pull-request-available
>
> In our cluster, each namespace has four NameNodes: one active, one standby,
> and two observers. When the standby NameNode performs a checkpoint, it
> transfer the fsimage to the other three NameNodes. However, we found that
> these transfer are performed serially.
> The reason is that the corePoolSize in ThreadPoolExecutor is 0, and the
> transfer task does not fill the LinkedBlockingQueue, resulting in only one
> thread transfer the fsimage at a time. This greatly increases the checkpoint
> time.
> {code:java}
> ExecutorService executor = new ThreadPoolExecutor(0,
> activeNNAddresses.size(), 100,
> TimeUnit.MILLISECONDS, new
> LinkedBlockingQueue<Runnable>(activeNNAddresses.size()),
> uploadThreadFactory); {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
