Hi
I was able to complete the load when setting those parameters set hive.merge.orcfile.stripe.level = false; those parameter didn’t have impact set hive.merge.mapfiles=false; set hive.merge.mapredfiles=false; I have an a similar script that load to similar table and this script doesn’t use this parameter and still can load hundreds of files. Any clue why it not consistent ? It may be a matter of some limit ? Thanks Yehuda *From:* Yehuda Finkelstein [mailto:yeh...@veracity-group.com] *Sent:* Tuesday, September 08, 2015 6:12 PM *To:* 'Prasanth Jayachandran'; 'user@hive.apache.org'; 'user@hive.apache.org ' *Cc:* Daniel Haviv *Subject:* RE: Error: java.lang.IllegalArgumentE:Column has wrong number of index entries found - when trying to insert from JSON external table to ORC table The hive version is 0.14 I will try to use the setting you mention. Thanks Yehuda *From:* Prasanth Jayachandran [mailto:pjayachand...@hortonworks.com <pjayachand...@hortonworks.com>] *Sent:* Tuesday, September 08, 2015 5:48 PM *To:* user@hive.apache.org; user@hive.apache.org *Cc:* Daniel Haviv; yeh...@veracity-group.com *Subject:* Re: Error: java.lang.IllegalArgumentE:Column has wrong number of index entries found - when trying to insert from JSON external table to ORC table What hive version are you using? Can you disable automerging of files and see if that works? set hive.merge.orcfile.stripe.level to false. Also set hive.merge.mapfiles to false and hive.merge.mapredfiles to false so that slow merging is also disabled. On Tue, Sep 8, 2015 at 5:29 AM -0700, "Yehuda Finkelstein" < yeh...@veracity-group.com> wrote: Hi all, We have a process that load data from folder that have json files. We use external table on top of the json folder and then insert into ORC table. We get the error below during the load “Column has wrong number of index entries found: 0 expected: 20” I validate the JSON data and it look ok. I notice that I got this error only when I reach amount of file (>40). I was able to complete the insert when the number of files is less than 40. I was able to load the data after concatenate all files into one big file. We are using this method for almost a year and never get this error before, even while we load folder with thousands of files. Task ID: task_1439882549481_118182_m_000000 URL: http://hdname:8088/taskdetails.jsp?jobid=job_1439882549481_118182&tipid=task_1439882549481_118182_m_000000 ----- Diagnostic Messages for this Task: Error: java.lang.IllegalArgumentException: Column has wrong number of index entries found: 0 expected: 20 at org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:726) at org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1614) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1996) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2288) at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:215) at org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.io.merge.MergeFileTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Cumulative CPU: 55.43 sec HDFS Read: 266399137 HDFS Write: 17963167 SUCCESS Stage-Stage-2: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 55 seconds 430 msec Regard, Yehuda *Yehuda Finkelshtein* *Senior DBA consultant*, Veracity Ltd. yeh...@veracity-group.com www.veracity-group.com [image: Description: Description: cid:image001.gif@01C8A535.5F5A43E0]
