Try registering your MyObject[] with Kryo.
On 25 Mar 2015 13:17, "donhoff_h" <165612...@qq.com> wrote:
> Hi, experts
>
> I wrote a very simple spark program to test the KryoSerialization
> function. The codes are as following:
>
> object TestKryoSerialization {
> def main(args: Array[String]) {
> val conf = new SparkConf()
> conf.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
> conf.set("spark.kryo.registrationRequired","true") //I use this
> statement to force checking registration.
> conf.registerKryoClasses(Array(classOf[MyObject]))
>
> val sc = new SparkContext(conf)
> val rdd =
> sc.textFile("hdfs://dhao.hrb:8020/user/spark/tstfiles/charset/A_utf8.txt")
> val objs = rdd.map(new MyObject(_,1)).collect()
> for (x <- objs ) {
> x.printMyObject
> }
> }
>
> The class MyObject is also a very simple Class, which is only used to test
> the serialization function:
> class MyObject {
> var myStr : String = ""
> var myInt : Int = 0
> def this(inStr : String, inInt : Int) {
> this()
> this.myStr = inStr
> this.myInt = inInt
> }
> def printMyObject {
> println("MyString is : "+myStr+"\tMyInt is : "+myInt)
> }
> }
>
> But when I ran the application, it reported the following error:
> java.lang.IllegalArgumentException: Class is not registered:
> dhao.test.Serialization.MyObject[]
> Note: To register this class use:
> kryo.register(dhao.test.Serialization.MyObject[].class);
> at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442)
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
> at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:565)
> at
> org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:161)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> I don't understand what cause this problem. I have used the
> "conf.registerKryoClasses" to register my class. Could anyone help me ?
> Thanks
>
> By the way, the spark version is 1.3.0.
>
>
