Re: Kryo serializer slower than Java serializer for Spark Streaming

2016-10-06 Thread Rajkiran Rajkumar
Oops, realized that I didn't reply to all. Pasting snippet again. Hi Sean, Thanks for the reply. I've done the part of forcing registration of classes to the kryo serializer. The observation is in that scenario. To give a sense of the data, they are records which are serialized using thrift and re

Re: Kryo serializer slower than Java serializer for Spark Streaming

2016-10-06 Thread Sean Owen
It depends a lot on your data. If it's a lot of custom types then Kryo doesn't have a lot of advantage, although, you want to make sure to register all your classes with kryo (and consider setting the flag that requires kryo registration to ensure it) because that can let kryo avoid writing a bunch