[
https://issues.apache.org/jira/browse/HDFS-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-8761.
---------------------------------
Resolution: Not A Problem
Hello [~odelalleau].
I answered your question on Stack Overflow. I'm pasting the answer here too.
After using the techniques I described to configure a path with a drive spec, I
expect you won't see these errors anymore. In the future, the best forum for
questions like this is the [email protected] mailing list.
You can specify a drive spec in {{hadoop.tmp.dir}} in core-site.xml by
prepending a '/' in front of the absolute path, and using '/' as the path
separator instead of '\' for all path elements. For example, if the desired
absolute path is D:\tmp\hdp, then it would look like this:
{code}
<property>
<name>hadoop.tmp.dir</name>
<value>/D:/tmp/hadoop</value>
</property>
{code}
The reason this works is that the default values for many of the HDFS
directories are configured to be file://${hadoop.tmp.dir}/suffix. See the
default definitions of {{dfs.namenode.name.dir}}, {{dfs.datanode.data.dir}} and
{{dfs.namenode.checkpoint.dir}} here:
http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Substituting the above value for {{hadoop.tmp.dir}} yields a valid {{file:}}
URL with a drive spec and no authority, which satisfies the requirements for
the HDFS configuration. It's important to use '/' instead of '\', because a
bare unencoded '\' character is not valid in URL syntax.
http://www.ietf.org/rfc/rfc1738.txt
If you prefer not to rely on this substitution behavior, then it's also valid
to override all configuration properties that make use of {{hadoop.tmp.dir}}
within your hdfs-site.xml file. Each value must be a full {{file:}} URL. For
example:
{code}
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///D:/tmp/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///D:/tmp/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///D:/tmp/hadoop/dfs/namesecondary</value>
</property>
{code}
You might find this more readable overall.
> Windows HDFS daemon - datanode.DirectoryScanner: Error compiling report (...)
> XXX is not a prefix of YYY
> --------------------------------------------------------------------------------------------------------
>
> Key: HDFS-8761
> URL: https://issues.apache.org/jira/browse/HDFS-8761
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
> Affects Versions: 2.7.1
> Environment: Windows 7, Java SDK 1.8.0_45
> Reporter: Olivier Delalleau
> Priority: Minor
>
> I'm periodically seeing errors like the one below output by the HDFS daemon
> (started with start-dfs.cmd). This is with the default settings for data
> location (=not specified in my hdfs-site.xml). I assume it may be fixable by
> specifying a path with the drive letter in the config file, however I haven't
> be able to do that (see
> http://stackoverflow.com/questions/31353226/setting-hadoop-tmp-dir-on-windows-gives-error-uri-has-an-authority-component).
> 15/07/11 17:29:57 ERROR datanode.DirectoryScanner: Error compiling report
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> \tmp\hadoop-odelalleau\dfs\data is not a prefix of
> D:\tmp\hadoop-odelalleau\dfs\data\current\BP-1474392971-10.128.22.110-1436634926842\current\finalized\subdir0\subdir0\blk_1073741825
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:566)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:425)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:406)
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:362)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
