Thanks a lot I'll try it out Daniel
> On 12 בדצמ׳ 2014, at 03:45, Prasanth Jayachandran > <pjayachand...@hortonworks.com> wrote: > > Thanks Daniel for filing the jira and the test case. I have put up a patch in > HIVE-9067 jira that should fix this issue. > > - Prasanth > > >> On Thu, Dec 11, 2014 at 3:29 AM, Daniel Haviv >> <daniel.ha...@veracity-group.com> wrote: >> Hi, >> I've created a JIRA with a test case: >> https://issues.apache.org/jira/browse/HIVE-9080 >> >> Thanks! >> Daniel >> >>> On Thu, Dec 11, 2014 at 12:49 AM, Prasanth Jayachandran >>> <pjayachand...@hortonworks.com> wrote: >>> I am unable to reproduce the case that causes exception that you are >>> seeing. Will be great if you can provide a repro. >>> >>> - Prasanth >>> >>> >>>> On Wed, Dec 10, 2014 at 1:43 PM, Prasanth Jayachandran >>>> <pjayachand...@hortonworks.com> wrote: >>>> I can see a bug for the case 2 where orc index is disabled. I have created >>>> a jira to track that issue. >>>> https://issues.apache.org/jira/browse/HIVE-9067 >>>> >>>> I am not sure why does it fail in case 1 though. Can you create a jira >>>> with a reproducible case? I can take a look at it. >>>> >>>> - Prasanth >>>> >>>> >>>>> On Wed, Dec 10, 2014 at 10:37 AM, Daniel Haviv >>>>> <daniel.ha...@veracity-group.com> wrote: >>>>> I've made a little experiment and recreated the table with >>>>> 'orc.create.index'='FALSE' and now it fails on something else: >>>>> Error: java.io.IOException: >>>>> org.apache.hadoop.hive.ql.metadata.HiveException: >>>>> java.lang.ClassCastException: >>>>> org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$StringStatisticsImpl >>>>> cannot be cast to >>>>> org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$BooleanStatisticsImpl >>>>> at >>>>> org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.map(MergeFileMapper.java:115) >>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) >>>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) >>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) >>>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>> at >>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >>>>> >>>>> It seems that the concatenation feature needs more work.. >>>>> >>>>> Daniel >>>>> >>>>>> On Wed, Dec 10, 2014 at 4:54 PM, Daniel Haviv >>>>>> <daniel.ha...@veracity-group.com> wrote: >>>>>> Hi, >>>>>> I'm trying to use the new concatenate command merge small ORC files and >>>>>> file right away: >>>>>> >>>>>> alter table requests partition(day_ts=1418083200, hour_ts=1418151600) >>>>>> concatenate; >>>>>> >>>>>> Diagnostic Messages for this Task: >>>>>> Error: java.lang.IllegalArgumentException: Column has wrong number of >>>>>> index entries found: 0 expected: 1 >>>>>> at >>>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:726) >>>>>> at >>>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1614) >>>>>> at >>>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1996) >>>>>> at >>>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2288) >>>>>> at >>>>>> org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:215) >>>>>> at >>>>>> org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98) >>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) >>>>>> at >>>>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) >>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) >>>>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>> at >>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>>>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >>>>>> >>>>>> >>>>>> Is there some property I need to set for ORC to be able to support >>>>>> concatenation? >>>>>> >>>>>> Thanks, >>>>>> Daniel >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You.
