Re: Serialization Problem in Spark Program

quot;;; > *Send time:* Thursday, Mar 26, 2015 3:23 PM > *To:* "donhoff_h"<165612...@qq.com>; > *Cc:* "user"; > *Subject: * Re: Serialization Problem in Spark Program > > Try registering your MyObject[] with Kryo. > On 25 Mar 2015 13:17, "donhoff

Re: Serialization Problem in Spark Program

Try registering your MyObject[] with Kryo. On 25 Mar 2015 13:17, "donhoff_h" <165612...@qq.com> wrote: > Hi, experts > > I wrote a very simple spark program to test the KryoSerialization > function. The codes are as following: > > object TestKryoSerialization { > def main(args: Array[String]) {

Re: Serialization Problem in Spark Program

you also need to register *array*s of MyObject. so change: conf.registerKryoClasses(Array(classOf[MyObject])) to conf.registerKryoClasses(Array(classOf[MyObject], classOf[Array[MyObject]])) On Wed, Mar 25, 2015 at 2:44 AM, donhoff_h <165612...@qq.com> wrote: > Hi, experts > > I wrote a very

Re: Serialization problem in Spark

I encounter the same problem with hadoop.fs.Configuration (very complex, unserializable class) basically if your closure contains any instance (not constant object/singleton! they are in the jar, not closure) that doesn't inherit Serializable, or their properties doesn't inherit Serializable, you a

Re: Serialization problem in Spark

did you try to register the class in Kryo serializer? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Mon, Jun 23, 2014 at 7:00 PM, rrussell25 wrote: > Thanks for pointer...tried Kryo and ran into a strange error: > > o

Re: Serialization problem in Spark

Thanks for pointer...tried Kryo and ran into a strange error: org.apache.spark.SparkException: Job aborted due to stage failure: Exception while deserializing and fetching task: com.esotericsoftware.kryo.KryoException: Unable to find class: rg.apache.hadoop.hbase.io.ImmutableBytesWritable It is s

Re: Serialization problem in Spark

I'm not sure if this is a Hadoop-centric issue or not. I had similar issues with non-serializable external library classes. I used a Kryo config (as illustrated here ) and registered the one troublesome class. It seemed to wor

Re: Serialization problem in Spark

Where are you getting serialization error. Its likely to be a different problem. Which class is not getting serialized? Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Thu, Jun 5, 2014 at 6:32 PM, Vibhor Banga wrote: >

Re: Serialization problem in Spark

Any inputs on this will be helpful. Thanks, -Vibhor On Thu, Jun 5, 2014 at 3:41 PM, Vibhor Banga wrote: > Hi, > > I am trying to do something like following in Spark: > > JavaPairRDD eventRDD = hBaseRDD.map(new > PairFunction, byte[], MyObject >() { > @Override > public