So, there is no way possible right now for Spark to read Hive 2.x data ? On Thu, Aug 24, 2017 at 12:17 AM, Eugene Koifman <ekoif...@hortonworks.com> wrote:
> This looks like you have some data written by Hive 2.x and Hive 1.x code > trying to read it. > > That is not supported. > > > > *From: *Aviral Agarwal <aviral12...@gmail.com> > *Reply-To: *"user@hive.apache.org" <user@hive.apache.org> > *Date: *Wednesday, August 23, 2017 at 12:24 AM > *To: *"user@hive.apache.org" <user@hive.apache.org> > *Subject: *Re: ORC Transaction Table - Spark > > > > Hi, > > Yes it caused by wrong naming convention of the delta directory : > > /apps/hive/warehouse/foo.db/bar/year=2017/month=5/delta_ > 0645253_0645253_0001 > > How do I solve this ? > > Thanks ! > Aviral Agarwal > > > > On Tue, Aug 22, 2017 at 11:50 PM, Eugene Koifman <ekoif...@hortonworks.com> > wrote: > > Could you do recursive “ls” in your table or partition that you are trying > to read? > > Most likely you have files that don’t follow expected naming convention > > > > Eugene > > > > > > *From: *Aviral Agarwal <aviral12...@gmail.com> > *Reply-To: *"user@hive.apache.org" <user@hive.apache.org> > *Date: *Tuesday, August 22, 2017 at 5:39 AM > *To: *"user@hive.apache.org" <user@hive.apache.org> > *Subject: *ORC Transaction Table - Spark > > > > Hi, > > > > I am trying to read hive orc transaction table through Spark but I am > getting the following error > > > Caused by: java.lang.RuntimeException: serious problem > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo( > OrcInputFormat.java:1021) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits( > OrcInputFormat.java:1048) > at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202) > ..... > Caused by: java.util.concurrent.ExecutionException: > java.lang.NumberFormatException: > For input string: "0645253_0001" > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo( > OrcInputFormat.java:998) > ... 118 more > > > Any help would be appreciated. > > Thanks and Regards, > Aviral Agarwal > > >
