Re: maprfs and spark libraries

As simple as that. Indeed, the spark jar i was linking to wasn't the mapr version. I just added spark-assembly-0.9.1-hadoop1.0.3-mapr-3.0.3.jar to the lib directory of my project as a unmanaged dependency for sbt. Thank you Cafe au Lait and to all of you guys. Regards, Nelson. -- View this mess

Re: maprfs and spark libraries

hi On May 26, 2014, at 06:48 PM, nelson wrote: The test application is built using sbt with the following dependency: - org.apache.spark spark-core 0.9.1   you need to remove this dependency. sbt will pack non-mapr version of hadoop classes. 

Re: maprfs and spark libraries

When I have stack traces, I usually see the MapR versions of the various hadoop classes, though maybe that's at a deeper level of the stack trace. If my memory is right though, this may point to the classpath having regular hadoop jars before the standard hadoop jars. My guess is that this is on t

Re: maprfs and spark libraries

I am using standard readers and writers i believe. When i locally run the app, spark is able to write on hdfs. Then i assume accessing and reading mfs is doable. Here is the piece of code i use for testing: /val list = List ("dad", "mum", "brother" , "sister") val mlist = sc.parallelize(list) mli

Re: maprfs and spark libraries

We use the mapr rpm and have successfully read and written hdfs data. Are you using custom readers/writers? Maybe the relevant stacktrace might help. Maybe also try a standard text reader and writer to see if there is a basic issue with accessing mfs? -Suren On Mon, May 26, 2014 at 11:31 AM,

Re: maprfs and spark libraries

thanks for replying guys. Mayur: Indeed i tried the local mode (sparkmaster: local[5]) before and the application runs well, no serialization problem. The problem arises as soon as i try to run the app over the cluster. Surendranauth: I just double checked my spark_classpath from spark_env.sh and

Re: maprfs and spark libraries

My team is successfully running on Spark on MapR. However, we add the mapr jars to the SPARK_CLASSAPTH on the workers, as well as making sure they are on the classpath of the driver. I'm not sure if we need every jar that we currently add but below is what we currently use. The important file in

Re: maprfs and spark libraries

Did you try in standalone mode. You may not see serialization issues in local threaded mode. Serialization errors are unlikely to be cause of Mapr hadoop version. Regards Mayur Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi