Erro:
2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException:
java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288)
... 11 more
Caused by: java.io.IOException: Malformed ORC file
hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001.
Invalid postscript.
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292)
at
org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 16 more
Hive Version: 0.13
Query:
SELECT t1.ID_TP_CLIENTE
, t1.COD_CLIENTE
, t1.DESC_MARCA
, t1.DOCUMENT_ID_NR
, t1.NM_CLIENTE
, t1.DT_NASCIMENTO
, t1.ID_TP_SEXO
, t1.DT_CRIACAO_CARGA
, t1.DT_LAST_CARGA
--
FROM ds_customer_qg t1
join ds_customer_qg t2
on t1.COD_CLIENTE = t2.COD_CLIENTE
De: Prasanth Jayachandran [mailto:pjayachand...@hortonworks.com]
Enviada em: Wednesday, October 1, 2014 3:24 PM
Para: user@hive.apache.org
Assunto: Re: Join error with ORC Hive tables
Hi
Can you post the exception stacktrace from hadoop execution logs? What version
of hive are you using?
Can you provide the join query that you are using?
Thanks
Prasanth Jayachandran
On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento
<thiago.be...@b2wdigital.com<mailto:thiago.be...@b2wdigital.com>> wrote:
Hi!
I'm trying to run a simple join with two tables stored as ORC File. (the same
query runs normally without ORC File)
I got this error while map-reduce job:
Driver returned: 2. Errors: OK
Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
Total jobs = 1
Stage-1 is selected by condition resolver.
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 12
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1411750973876_0099, Tracking URL =
http://arnaldo02:8088/proxy/application_1411750973876_0099/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
Hadoop job information for Stage-1: number of mappers: 42; number of reducers:
12
2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88
sec
2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93
sec
2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04
sec
2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02
sec
2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41
sec
2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71
sec
2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69
sec
2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43
sec
2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3
sec
2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77
sec
2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87
sec
2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72
sec
2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83
sec
2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45
sec
2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46
sec
2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74
sec
2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51
sec
2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57
sec
2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95
sec
2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59
sec
2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89
sec
2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72
sec
2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76
sec
2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87
sec
2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71
sec
2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66
sec
2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7
sec
2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91
sec
2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35
sec
2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34
sec
2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69
sec
2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38
sec
2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28
sec
2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39
sec
2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79
sec
2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94
sec
2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31
sec
2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21
sec
2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94
sec
2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2
sec
2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7
sec
2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75
sec
2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35
sec
2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24
sec
2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35
sec
2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34
sec
2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81
sec
2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71
sec
2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74
sec
2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6
sec
2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18
sec
2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43
sec
2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71
sec
2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
459.27 sec
MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
Ended Job = job_1411750973876_0099 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843
HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
Any idea?
Tks,
Thiago Henrique S. Bento
B2W Digital / B2W Labs | BIT SP
thiago.be...@b2wdigital.com<mailto:thiago.be...@b2wdigital.com>
Tel.: +55 (11) 3454-3024
.............................................................................................................................................................
B2W DIGITAL #Americanas.com<http://americanas.com/>
#Shoptime.com<http://shoptime.com/> #Submarino.com<http://submarino.com/>
#SouBarato.com<http://soubarato.com/>
________________________________
Esta mensagem pode conter informações confidenciais e somente o indivíduo ou
entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da
mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem
tenha sido recebida por engano, solicitamos que o fato seja comunicado ao
remetente e que a mensagem seja eliminada de seu sistema imediatamente. É
vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir
ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a
monitoramento.
This message may include confidential information and only the intended
addresses have the right to use it as is, or any part of it. A wrong
transmission does not break its confidentiality. If you've received it because
of a mistake or erroneous transmission, please notify the sender and delete it
from your system immediately. This communication environment is controlled and
monitored.
B2W Digital
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
