Did you enable Kryo and have it use your registrator using spark.serializer=org.apache.spark.serializer.KryoSerializer and spark.kryo.registrator=mypackage.MyRegistrator ? It looks like the serializer being used is the default Java one
http://spark.apache.org/docs/latest/tuning.html#data-serialization On Fri, Aug 1, 2014 at 1:23 AM, gpatcham <gpatc...@gmail.com> wrote: > I'm new to spark programming and here I'm trying to use third party class > in > map with kryo serializer > > val deviceApi = new DeviceApi() > > > deviceApi.loadDataFromStream(this.getClass.getClassLoader.getResourceAsStream("20140730.json")) > val properties = uaRDD1.map(line => deviceApi.getProperties(line) > > Here is kryo register code > import mobi.mtld.da.device.DeviceApi > > class MyRegistrator extends KryoRegistrator { > override def registerClasses(kryo: Kryo) { > kryo.register(classOf[DeviceApi]) > } > } > > I'm still seeing this error > > Caused by: org.apache.spark.SparkException: Job aborted: Task not > serializable: java.io.NotSerializableException: > mobi.mtld.da.device.DeviceApi > > > > Can anyone please let me know if something is wrong > > Thanks > Giri > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Issue-using-kryo-serilization-tp11129.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
