Hello,
I have been using Spark for quite some time, and I now get this error
(please stderr output below) when accessing hdfs. It seems to come from
Hadoop, however, I can access hdfs from the command line without any
problem.
The WARN on the first seems to be key, because it never appeared
previously. My HADOOP_CONF_DIR is correctly set, so I don't know why
libhadoop cannot be loaded.
Please note that this " java.net.UnknownHostException: crm" is rather
strange because crm is not an host, it is the name of my hadoop cluster,
as specified in hadoop core.xml file in dfs.nameservices and fs.defaultFS
hadoop properties.
Thanks for your help.
14/09/17 22:33:49 WARN BlockReaderLocal: The short-circuit local reads
feature cannot
be used because libhadoop cannot be loaded.
14/09/17 22:33:49 WARN BlockReaderLocal: The short-circuit local reads
feature cannot
be used because libhadoop cannot be loaded.
14/09/17 22:34:01 WARN TaskSetManager: Lost TID 20 (task 1.0:20)
14/09/17 22:34:01 WARN TaskSetManager: Loss was due to
java.lang.IllegalArgumentExcep
tion
java.lang.IllegalArgumentException: java.net.UnknownHostException: crm
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.jav
a:418)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.ja
va:231)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:13
9)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSys
tem.java:136)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:166)
at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:653)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.jav
a:389)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.jav
a:362)
at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
at
org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala
