Hi, all
I wrote a spark program which uses the Kryo serialization. When I count a rdd
which type is RDD[(String,String)], it reported an Exception like the following
:
* Class is not registered: org.apache.spark.util.collection.CompactBuffer[]
* Note: To register this class use:
kryo.register(org.apache.spark.util.collection.CompactBuffer[].class);
I found that CompactBuffer is a private class that I could not use the API
sparkConf.registerKryoClasses to register it. Although I found a solution from
Andras Barjak that is using the statement :
kryo.register(ClassTag(Class.forName("org.apache.spark.util.collection.CompactBuffer")).wrap.runtimeClass),
I tried it with sparkConfg.registerKryoClasses, and only got another Exception
:
* Class is not registered: scala.reflect.ManifestFactory$$anon$1
Could anybody tell me :
1.Why should I register CompactBuffer[] ? My program never used it.
2.How to solve this problem?
Many Thanks!
