I finally managed to track down the source of the kryo issues that I was
having under mesos.
What happens is the for a reason that I haven't tracked down yet, a handful
of the scala collection classes from chill-scala down get registered by the
mesos executors, but they do all get registered in the driver process.
This led to scala.Some classes which were serialized by the executors being
incorrectly deserialized as scala.collections.Wrappers$SeqWrapper in driver
during task deserialization, causing a KryoException.
I resolved this issue in my spark job by explicitly registering the classes
in my Registrator like so:
kryo.register(scala.collection.convert.Wrappers.IteratorWrapper.class);
kryo.register(scala.collection.convert.Wrappers.SeqWrapper.class);
kryo.register(scala.collection.convert.Wrappers.MapWrapper.class);
kryo.register(scala.collection.convert.Wrappers.JListWrapper.class);
kryo.register(scala.collection.convert.Wrappers.JMapWrapper.class);
Again, I'm not sure why they don't get registered in the mesos executors,
but I wanted to report wht I found as well as a workaround in case anyone
else hit this (extraordinarily frustrating) issue again.
Some interactive debugging note are available in this gist:
https://gist.github.com/sorenmacbeth/28707a7a973f7a1982dc
Cheers,
Soren
