Re: Concatenating ORC files

2014-12-12 Thread Daniel Haviv
Thanks a lot I'll try it out Daniel > On 12 בדצמ׳ 2014, at 03:45, Prasanth Jayachandran > wrote: > > Thanks Daniel for filing the jira and the test case. I have put up a patch in > HIVE-9067 jira that should fix this issue. > > - Prasanth > > >> On Thu, Dec 11, 2014 at 3:29 AM, Daniel Hav

Re: Concatenating ORC files

2014-12-11 Thread Prasanth Jayachandran
Thanks Daniel for filing the jira and the test case. I have put up a patch in HIVE-9067 jira that should fix this issue.  - Prasanth On Thu, Dec 11, 2014 at 3:29 AM, Daniel Haviv wrote: > Hi, > I've created a JIRA with a test case: > https://issues.apache.org/jira/browse/HIVE-9080 > Thanks! >

Re: Concatenating ORC files

2014-12-11 Thread Daniel Haviv
Hi, I've created a JIRA with a test case: https://issues.apache.org/jira/browse/HIVE-9080 Thanks! Daniel On Thu, Dec 11, 2014 at 12:49 AM, Prasanth Jayachandran < pjayachand...@hortonworks.com> wrote: > I am unable to reproduce the case that causes exception that you are > seeing. Will be great

Re: Concatenating ORC files

2014-12-10 Thread Prasanth Jayachandran
I am unable to reproduce the case that causes exception that you are seeing. Will be great if you can provide a repro. - Prasanth On Wed, Dec 10, 2014 at 1:43 PM, Prasanth Jayachandran wrote: > I can see a bug for the case 2 where orc index is disabled. I have created a > jira to track that

Re: Concatenating ORC files

2014-12-10 Thread Prasanth Jayachandran
I can see a bug for the case 2 where orc index is disabled. I have created a jira to track that issue. https://issues.apache.org/jira/browse/HIVE-9067 I am not sure why does it fail in case 1 though. Can you create a jira with a reproducible case? I can take a look at it. - Prasanth On W

Re: Concatenating ORC files

2014-12-10 Thread Daniel Haviv
HI Prasanth, The first attempt had ("orc.compress"="Snappy") and all the files under it were created the same way so I'm assuming they all should have indexes created. In the second attempt I used ("orc.create.index"="false", "orc.compress"="Snappy"). Thanks, Daniel ll On Wed, Dec 10, 2014 at 9:

Re: Concatenating ORC files

2014-12-10 Thread Prasanth Jayachandran
Hi Daniel In you first run, are there some files with “orc.create.index”=“false”? What are the table properties used to create ORC files in both cases? - Prasanth On Wed, Dec 10, 2014 at 7:55 AM, Daniel Haviv wrote: > Hi, > I'm trying to use the new concatenate command merge small ORC fil

Re: Concatenating ORC files

2014-12-10 Thread Daniel Haviv
I've made a little experiment and recreated the table with 'orc.create.index'='FALSE' and now it fails on something else: Error: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$StringStatistic

Concatenating ORC files

2014-12-10 Thread Daniel Haviv
Hi, I'm trying to use the new concatenate command merge small ORC files and file right away: alter table requests partition(day_ts=1418083200, hour_ts=1418151600) concatenate; Diagnostic Messages for this Task: Error: java.lang.IllegalArgumentException: Column has wrong number of index entries f