I created the table with stored as ORC, and I did a sqoop job from oracle to load data to this table. This table have something like 65 million lines but when I load 10 million, the query runs normally. I think that is problems with memory, my cluster doesn't have much memory.
De: Prasanth Jayachandran [mailto:pjayachand...@hortonworks.com] Enviada em: Thursday, October 2, 2014 4:53 PM Para: user@hive.apache.org Assunto: Re: RES: Join error with ORC Hive tables Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript. >From the exception it looks like the file it is trying to read from table >stored as ORC but the above file is not an ORC file. How is the ORC table >loaded? My suspicion is some non-ORC file ended up in ORC table. Thanks Prasanth Jayachandran On Oct 2, 2014, at 12:26 PM, Thiago Henrique dos Santos Bento <thiago.be...@b2wdigital.com<mailto:thiago.be...@b2wdigital.com>> wrote: Erro: 2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288) ... 11 more Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript. at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292) at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65) ... 16 more Hive Version: 0.13 Query: SELECT t1.ID_TP_CLIENTE , t1.COD_CLIENTE , t1.DESC_MARCA , t1.DOCUMENT_ID_NR , t1.NM_CLIENTE , t1.DT_NASCIMENTO , t1.ID_TP_SEXO , t1.DT_CRIACAO_CARGA , t1.DT_LAST_CARGA -- FROM ds_customer_qg t1 join ds_customer_qg t2 on t1.COD_CLIENTE = t2.COD_CLIENTE De: Prasanth Jayachandran [mailto:pjayachand...@hortonworks.com] Enviada em: Wednesday, October 1, 2014 3:24 PM Para: user@hive.apache.org<mailto:user@hive.apache.org> Assunto: Re: Join error with ORC Hive tables Hi Can you post the exception stacktrace from hadoop execution logs? What version of hive are you using? Can you provide the join query that you are using? Thanks Prasanth Jayachandran On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <thiago.be...@b2wdigital.com<mailto:thiago.be...@b2wdigital.com>> wrote: Hi! I'm trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File) I got this error while map-reduce job: Driver returned: 2. Errors: OK Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab Total jobs = 1 Stage-1 is selected by condition resolver. Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 12 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/ Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099 Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12 2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0% 2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec 2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec 2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec 2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec 2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec 2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec 2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec 2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec 2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec 2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec 2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec 2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec 2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec 2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec 2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec 2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec 2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec 2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec 2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec 2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec 2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec 2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec 2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec 2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec 2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec 2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec 2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec 2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec 2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec 2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec 2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec 2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec 2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec 2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec 2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec 2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec 2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec 2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec 2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec 2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec 2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec 2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec 2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec 2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec 2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec 2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec 2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec 2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec 2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec 2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec 2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec 2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec 2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec 2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec 2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec 2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec 2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec 2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec 2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec 2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec Ended Job = job_1411750973876_0099 with errors FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec Any idea? Tks, Thiago Henrique S. Bento B2W Digital / B2W Labs | BIT SP thiago.be...@b2wdigital.com<mailto:thiago.be...@b2wdigital.com> Tel.: +55 (11) 3454-3024 ............................................................................................................................................................. B2W DIGITAL #Americanas.com<http://americanas.com/> #Shoptime.com<http://shoptime.com/> #Submarino.com<http://submarino.com/> #SouBarato.com<http://soubarato.com/> ________________________________ Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento. This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored. B2W Digital CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
