I guess I already have the answer of what I have to do here, which is to configure the kryo object with the strategy as above. Now the question becomes: how can I pass this custom kryo configuration to the spark kryo serializer / kryo registrator? I've had a look at the code but I am still fairly new to Scala and I can't see how I would do this. In the worst case, could I override the newKryo method and put my configuration there? It appears to me that method is the one where the kryo instance is created.
Simone Franzini, PhD http://www.linkedin.com/in/simonefranzini On Tue, Nov 25, 2014 at 2:38 PM, Simone Franzini <captainfr...@gmail.com> wrote: > I am running into the following NullPointerException: > > com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException > Serialization trace: > underlying (scala.collection.convert.Wrappers$JListWrapper) > myArrayField (MyCaseClass) > at > com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > > I have been running into similar issues when using avro classes, that I > was able to resolve by registering them with a Kryo serializer that uses > chill-avro. However, in this case the field is in a case class and it seems > that registering the class does not help. > > I found this stack overflow that seems to be relevant: > > http://stackoverflow.com/questions/23962796/kryo-readobject-cause-nullpointerexception-with-arraylist > I have this line of code translated to Scala, that supposedly solves the > issue: > > val kryo = new Kryo() > kryo.getInstantiatorStrategy().asInstanceOf[Kryo.DefaultInstantiatorStrategy].setFallbackInstantiatorStrategy(new > StdInstantiatorStrategy()) > > However, I am not sure where this line should be placed to take effect. > > I already have the following, should it go somewhere in here? > class MyRegistrator extends KryoRegistrator { > override def registerClasses(kryo: Kryo) { > kryo.register(...) > } > } > > > Simone Franzini, PhD > > http://www.linkedin.com/in/simonefranzini >
