Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

*I am facing similar issue to Spark-3447 with spark streaming Api, Kryo Serializer, Avro messages. If avro message is simple, its fine. but if the avro message has Union/Arrays its failing with the exception Below:* ERROR scheduler.JobScheduler: Error running job streaming job 1411043845000 ms.0 or

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Added some more info on this issue in the tracker Spark-3447 https://issues.apache.org/jira/browse/SPARK-3447 - Thanks & Regards, Mohan -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-fails-with-avro-having-Arrays-and-unions-but-succeeds-with-simple

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Hi frank, thanks for the info, thats great. but im not saying Avro serializer is failing. Kryo is failing but im using kryo serializer. and registering Avro generated classes with kryo. sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); sparkConf.set("spark.kr

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Thanks for the info frank. so your suggestion could be to use Avro serializer. i just have to configure it like Kryo for the same property? and is there any registering process for this or just specify serializer? Also does it effect performance. what measures to be taken to avoid. (im using kryo

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Thanks for the info frank. Twitter's-chill avro serializer looks great. But how does spark identifies it as serializer, as its not extending from KryoSerializer. (sorry scala is an alien lang for me). - Thanks & Regards, Mohan -- View this message in context: http://apache-spark-user-list.