maobaolong created HDFS-13269:
---------------------------------
Summary: After too many open file exception occurred, the standby
NN never do checkpoint
Key: HDFS-13269
URL: https://issues.apache.org/jira/browse/HDFS-13269
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.2.0
Reporter: maobaolong
do saveNameSpace in dfsadmin.
The output as following:
{code:java}
saveNamespace: No image directories available!
{code}
The Namenode log show:
{code:java}
[2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] :
Triggering checkpoint because there have been 10159265 txns since the last
checkpoint, which exceeds the configured threshold 10000000
[2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] : Save
namespace ...
...
[2018-01-13T10:37:10.539+08:00] [WARN] [1985938863@qtp-61073295-1 - Acceptor0
HttpServer2$SelectChannelConnectorWithSafeStartup@HOST_A:50070] : EXCEPTION
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at
org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
[2018-01-13T10:37:15.421+08:00] [ERROR] [FSImageSaver for /data0/nn of type
IMAGE_AND_EDITS] : Unable to save image for /data0/nn
java.io.FileNotFoundException:
/data0/nn/current/fsimage_0000000040247283317.md5.tmp (Too many open files)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at
org.apache.hadoop.hdfs.util.AtomicFileOutputStream.<init>(AtomicFileOutputStream.java:58)
at
org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:157)
at
org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:149)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:990)
at
org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:1039)
at java.lang.Thread.run(Thread.java:745)
[2018-01-13T10:37:15.421+08:00] [ERROR] [Standby State Checkpointer] : Error
reported on storage directory Storage Directory /data0/nn
[2018-01-13T10:37:15.421+08:00] [WARN] [Standby State Checkpointer] : About to
remove corresponding storage: /data0/nn
[2018-01-13T10:37:15.429+08:00] [ERROR] [Standby State Checkpointer] :
Exception in doCheckpoint
java.io.IOException: Failed to save in any storage directories while saving
namespace.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1176)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:353)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$700(StandbyCheckpointer.java:260)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:280)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:276)
...
[2018-01-13T15:52:33.783+08:00] [INFO] [Standby State Checkpointer] : Save
namespace ...
[2018-01-13T15:52:33.783+08:00] [ERROR] [Standby State Checkpointer] :
Exception in doCheckpoint
java.io.IOException: No image directories available!
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1152)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:353)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$700(StandbyCheckpointer.java:260)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:280)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:276)
{code}
After seeing the NNStorage#reportErrorsOnDirectory
{code:java}
private void reportErrorsOnDirectory(StorageDirectory sd) {
LOG.error("Error reported on storage directory {}", sd);
if(LOG.isDebugEnabled()){
String lsd = listStorageDirectories();
LOG.debug("current list of storage dirs:{}", lsd);
}
LOG.warn("About to remove corresponding storage: {}", sd.getRoot()
.getAbsolutePath());
try {
sd.unlock();
} catch (Exception e) {
LOG.warn("Unable to unlock bad storage directory: {}", sd.getRoot()
.getPath(), e);
}
if (getStorageDirs().remove(sd)) {
this.removedStorageDirs.add(sd);
}
if(LOG.isDebugEnabled()){
String lsd = listStorageDirectories();
LOG.debug("at the end current list of storage dirs:{}", lsd);
}
}
{code}
I think when the FileNotFoundException(Too many open file) occurred, we should
not remove the storageDirs, because the exception can be cure after several
minute.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
