HI Prasanth,
The first attempt had ("orc.compress"="Snappy") and all the files under it
were created the same way so I'm assuming they all should have indexes
created.
In the second attempt I used ("orc.create.index"="false",
"orc.compress"="Snappy").Thanks, Daniel ll On Wed, Dec 10, 2014 at 9:04 PM, Prasanth Jayachandran < pjayachand...@hortonworks.com> wrote: > Hi Daniel > > In you first run, are there some files with “orc.create.index”=“false”? > What are the table properties used to create ORC files in both cases? > > - Prasanth > > > On Wed, Dec 10, 2014 at 7:55 AM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> I'm trying to use the new concatenate command merge small ORC files and >> file right away: >> >> alter table requests partition(day_ts=1418083200, hour_ts=1418151600) >> concatenate; >> >> Diagnostic Messages for this Task: >> Error: java.lang.IllegalArgumentException: Column has wrong number of >> index entries found: 0 expected: 1 >> at >> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:726) >> at >> org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1614) >> at >> org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1996) >> at >> org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2288) >> at >> org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:215) >> at >> org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98) >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) >> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >> >> >> Is there some property I need to set for ORC to be able to support >> concatenation? >> >> Thanks, >> Daniel >> >> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
