Istvan Fajth created HDFS-15303:
-----------------------------------
Summary: Provide a tool that can validate/fix the block file
placement in DataNode data directories
Key: HDFS-15303
URL: https://issues.apache.org/jira/browse/HDFS-15303
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Istvan Fajth
We recently run into an issue where during an upgrade from Hadoop2 to Hadoop3,
the filesystem under DataNode data directories was left in an intermediate
state, and part of the directories were in Hadoop2 format other parts in
Hadoop3 format.
At first we had to rollback the upgrade, and after the rollback we started to
see FileNotFoundExceptions for particular block files.
The exception was logged on the DataNodes and failed the jobs as well
sometimes. HDFS NameNode did not show any missing blocks, and we found the
block files and meta files also in the DataNode's data directories, but at a
different location.
This was the point when we realized that something went wrong during the
rollback, and some of the data directories had blocks placed according to
Hadoop3 rules, while other were placed according to Hadoop2 rules. We suspect a
possible premature DataNode shutdown or an unknown failure during the rollback,
but at the point when we realized what is the issue and could check into
things, we already ran out of the logs that would have been able to show us the
cause.
This JIRA is to suggest two new commands that can help administrators in this
situation, to validate the data directory and ensure that blocks are placed
correctly according to the rules, and to fix the data directory layout if
needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
