Re: ORC Transaction Table - Spark

2017-08-24 Thread Gopal Vijayaraghavan
> Or, is this an artifact of an incompatibility between ORC files written by > the Hive 2.x ORC serde not being readable by the Hive 1.x ORC serde? > 3. Is there a difference in the ORC file format spec. at play here? Nope, we're still defaulting to hive-0.12 format ORC files in Hive-2.x. We h

RE: ORC Transaction Table - Spark

2017-08-24 Thread Larson, Kurt
serde? 6. Any similar issues with Parquet format in Hive 1.x and 2.x? From: Aviral Agarwal [mailto:aviral12...@gmail.com] Sent: Wednesday, August 23, 2017 10:34 PM To: user@hive.apache.org Subject: Re: ORC Transaction Table - Spark So, there is no way possible right now for Spark to read

Re: ORC Transaction Table - Spark

2017-08-24 Thread Furcy Pin
r@hive.apache.org" >> *Date: *Wednesday, August 23, 2017 at 12:24 AM >> *To: *"user@hive.apache.org" >> *Subject: *Re: ORC Transaction Table - Spark >> >> >> >> Hi, >> >> Yes it caused by wrong naming convention of the delta direct

Re: ORC Transaction Table - Spark

2017-08-23 Thread Aviral Agarwal
ral Agarwal > *Reply-To: *"user@hive.apache.org" > *Date: *Wednesday, August 23, 2017 at 12:24 AM > *To: *"user@hive.apache.org" > *Subject: *Re: ORC Transaction Table - Spark > > > > Hi, > > Yes it caused by wrong naming convention of the delt

Re: ORC Transaction Table - Spark

2017-08-23 Thread Eugene Koifman
This looks like you have some data written by Hive 2.x and Hive 1.x code trying to read it. That is not supported. From: Aviral Agarwal Reply-To: "user@hive.apache.org" Date: Wednesday, August 23, 2017 at 12:24 AM To: "user@hive.apache.org" Subject: Re: ORC Transaction Tab

Re: ORC Transaction Table - Spark

2017-08-23 Thread Aviral Agarwal
Hi, Yes it caused by wrong naming convention of the delta directory : /apps/hive/warehouse/foo.db/bar/year=2017/month=5/delta_0645253_0645253_0001 How do I solve this ? Thanks ! Aviral Agarwal On Tue, Aug 22, 2017 at 11:50 PM, Eugene Koifman wrote: > Could you do recursive “ls” in your table

Re: ORC Transaction Table - Spark

2017-08-22 Thread Eugene Koifman
Could you do recursive “ls” in your table or partition that you are trying to read? Most likely you have files that don’t follow expected naming convention Eugene From: Aviral Agarwal Reply-To: "user@hive.apache.org" Date: Tuesday, August 22, 2017 at 5:39 AM To: "user@hive.apache.org" Subjec