The error is resolved. I was using a comparator which was not serialised
because of which it was throwing the error.
I have now switched to kryo serializer as it is faster than java serialser.
I have set the required config
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "MyRegistrator");
and also in MyRegistrator class I have registered all the classes I am
serialising.
How can I confirm that my code is actually using kryo serialiser and not
java serialiser now ?
PS : It seems like my code is still not using kryo serialiser.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Error-related-to-serialisation-in-spark-streaming-tp6801p6904.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
