When I have stack traces, I usually see the MapR versions of the various
hadoop classes, though maybe that's at a deeper level of the stack trace.
If my memory is right though, this may point to the classpath having
regular hadoop jars before the standard hadoop jars. My guess is that this
is on the driver side, if so.
I had to make sure to put the MapR jars as the very first entries on my
driver classpath.
-Suren
On Mon, May 26, 2014 at 12:11 PM, nelson <nelson.verd...@ysance.com> wrote:
> I am using standard readers and writers i believe.
> When i locally run the app, spark is able to write on hdfs. Then i assume
> accessing and reading mfs is doable.
>
> Here is the piece of code i use for testing:
> /val list = List ("dad", "mum", "brother" , "sister")
> val mlist = sc.parallelize(list)
> mlist.saveAsTextFile("maprfs:///user/nelson/test")/
>
> and the stack trace:
> /14/05/26 16:02:54 WARN scheduler.TaskSetManager: Loss was due to
> java.lang.NullPointerException
> java.lang.NullPointerException
> at org.apache.hadoop.fs.FileSystem.fixName(FileSystem.java:187)
> at
> org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:123)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:115)
> at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:617)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:439)
> at
>
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:412)
> at
> org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
> at
> org.apache.spark.SparkContext$$anonfun$15.apply(SparkContext.scala:391)
> at
>
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
> at
>
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:111)
> at scala.Option.map(Option.scala:145)
> at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:111)
> at
> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:154)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:149)
> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:64)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at
> org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
> at org.apache.spark.scheduler.Task.run(Task.scala:53)
> at
>
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:211)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
> at
>
> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at
> org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:176)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)/
>
> The uri name seems to be the issue now as i happen to get rid of the
> serialization issue.
>
> Regards,
> Nelson
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/maprfs-and-spark-libraries-tp6392p6402.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
--
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440 NINTH AVENUE, 11TH FLOOR
NEW YORK, NY 10001
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@v <suren.hira...@sociocast.com>elos.io
W: www.velos.io
