Hi, experts
I wrote a very simple spark program to test the KryoSerialization function. The
codes are as following:
object TestKryoSerialization {
def main(args: Array[String]) {
val conf = new SparkConf()
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.set("spark.kryo.registrationRequired","true") //I use this statement
to force checking registration.
conf.registerKryoClasses(Array(classOf[MyObject]))
val sc = new SparkContext(conf)
val rdd =
sc.textFile("hdfs://dhao.hrb:8020/user/spark/tstfiles/charset/A_utf8.txt")
val objs = rdd.map(new MyObject(_,1)).collect()
for (x <- objs ) {
x.printMyObject
}
}
The class MyObject is also a very simple Class, which is only used to test the
serialization function:
class MyObject {
var myStr : String = ""
var myInt : Int = 0
def this(inStr : String, inInt : Int) {
this()
this.myStr = inStr
this.myInt = inInt
}
def printMyObject {
println("MyString is : "+myStr+"\tMyInt is : "+myInt)
}
}
But when I ran the application, it reported the following error:
java.lang.IllegalArgumentException: Class is not registered:
dhao.test.Serialization.MyObject[]
Note: To register this class use:
kryo.register(dhao.test.Serialization.MyObject[].class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:161)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I don't understand what cause this problem. I have used the
"conf.registerKryoClasses" to register my class. Could anyone help me ? Thanks
By the way, the spark version is 1.3.0.