Re: Exception with SparkSql and Avro

2014-09-23 Thread Zalzberg, Idan (Agoda)
Thanks, I didn't create the tables myself as I have no control over that process. However these tables are read just fund using the Jdbc connection to the hiveserver2 so it should be possible On Sep 24, 2014 12:48 AM, Michael Armbrust wrote: Can you show me the DDL you are using? Here is an exa

Re: Exception with SparkSql and Avro

2014-09-23 Thread Michael Armbrust
Can you show me the DDL you are using? Here is an example of a way I got the avro serde to work: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala#L246 Also, this isn't ready for primetime yet, but a quick plug for some ongoing work: http

RE: Exception with SparkSql and Avro

2014-09-22 Thread Zalzberg, Idan (Agoda)
Hello, I am trying to read a hive table that is stored in Avro DEFLATE files. something simple like "SELECT * FROM X LIMIT 10" I get 2 exceptions in the logs: 2014-09-23 09:27:50,157 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 10.0 in stage 1.0 (TID 10, cl.local): org.apache.avro.A