Alexandre Linte created HIVE-15415:
--------------------------------------
Summary: Random "java.util.ConcurrentModificationException"
Key: HIVE-15415
URL: https://issues.apache.org/jira/browse/HIVE-15415
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 2.1.0
Environment: Hadoop 2.7.3, Hive 2.1.0
Reporter: Alexandre Linte
I'm regularly facing Hive job failures through Oozie or through the beeline
CLI. The jobs exit with an error "FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask.
java.util.ConcurrentModificationException (state=08S01,code=1)" but not 100% of
the time.
it's also important to underline that only one user is working on the table
when the jobs are running.
- stderr
{noformat}
Connecting to jdbc:hive2://hiveserver2.bigdata.fr:10000/default
Connected to: Apache Hive (version 2.1.0)
Driver: Hive JDBC (version 2.1.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
No rows affected (1.475 seconds)
No rows affected (0.004 seconds)
No rows affected (0.004 seconds)
No rows affected (58.977 seconds)
No rows affected (5.524 seconds)
No rows affected (5.235 seconds)
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask.
java.util.ConcurrentModificationException (state=08S01,code=1)
Closing: 0: jdbc:hive2://hiveserver2.bigdata.fr:10000/default
Intercepting System.exit(2)
{noformat}
- stdout
{noformat}
Beeline command arguments :
-u
jdbc:hive2://hiveserver2.bigdata.fr:10000/default
-n
my_user
-p
DUMMY
-d
org.apache.hive.jdbc.HiveDriver
-f
full_job
-a
delegationToken
--hiveconf
mapreduce.job.tags=oozie-75b060aacd7ec48c4ed637855e413280
Fetching child yarn jobs
tag id : oozie-75b060aacd7ec48c4ed637855e413280
Child yarn jobs are found -
=================================================================
>>> Invoking Beeline command line now >>>
0: jdbc:hive2://hiveserver2.bigdata.fr> use my_db;
0: jdbc:hive2://hiveserver2.bigdata.fr> set hive.execution.engine=tez;
0: jdbc:hive2://hiveserver2.bigdata.fr> set tez.queue.name=tez_queue;
0: jdbc:hive2://hiveserver2.bigdata.fr>
0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
^M_fd_livcfm
. . . . . . . . . . . . . . . . . . . . . . .> select
. . . . . . . . . . . . . . . . . . . . . . .> col.co_cd as co_cd,
. . . . . . . . . . . . . . . . . . . . . . .> col.line_co_cd as line_co_cd,
. . . . . . . . . . . . . . . . . . . . . . .>
unix_timestamp(min(tt.statut_dt)) ^M as statut_dt
. . . . . . . . . . . . . . . . . . . . . . .> from
dlk_scf_rn_customer_order_li ^Mne col
. . . . . . . . . . . . . . . . . . . . . . .> join
dlk_scf_rn_shipment_handling ^M_utility shu
. . . . . . . . . . . . . . . . . . . . . . .> on shu.co_cd =col.co_cd
. . . . . . . . . . . . . . . . . . . . . . .> and shu.line_co_cd =
col.line_co_ ^Mcd
. . . . . . . . . . . . . . . . . . . . . . .> join ( select
scaler_internal_ref ^M, statut_dt,recep_number,state,reason
. . . . . . . . . . . . . . . . . . . . . . .> from
dlk_scf_rn_transport_trackin ^Mg where state='LIV' and reason='CFM' ) tt
. . . . . . . . . . . . . . . . . . . . . . .> on
concat('CAL',shu.c_waybill_no) ^M =tt.scaler_internal_ref group by
col.co_cd,col.line_co_cd;
Heart beat
Heart beat
0: jdbc:hive2://hiveserver2.bigdata.fr>
0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
^M_fd_cae
. . . . . . . . . . . . . . . . . . . . . . .> select
. . . . . . . . . . . . . . . . . . . . . . .> po_cd as cae, line_po_cd as
lcae, ^M origin_co_cd, origin_line_co_cd
. . . . . . . . . . . . . . . . . . . . . . .> from
dlk_scf_rn_purchase_order_li ^Mne
. . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"7")=1;
0: jdbc:hive2://hiveserver2.bigdata.fr>
0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
^M_fd_cai
. . . . . . . . . . . . . . . . . . . . . . .> select
. . . . . . . . . . . . . . . . . . . . . . .> po_cd as cai, line_po_cd as
lcai, ^M origin_co_cd, origin_line_co_cd
. . . . . . . . . . . . . . . . . . . . . . .> from
dlk_scf_rn_purchase_order_li ^Mne
. . . . . . . . . . . . . . . . . . . . . . .> where instr(po_cd,"L")=1;
0: jdbc:hive2://hiveserver2.bigdata.fr>
0: jdbc:hive2://hiveserver2.bigdata.fr> insert overwrite table main_table
^M_fd_cce
. . . . . . . . . . . . . . . . . . . . . . .> select co_cd as cce,line_co_cd
as ^M lcce,demand_order_ref1,
. . . . . . . . . . . . . . . . . . . . . . .> case when demand_order_ref2=''
. . . . . . . . . . . . . . . . . . . . . . .> then 1
. . . . . . . . . . . . . . . . . . . . . . .> else demand_order_ref2
. . . . . . . . . . . . . . . . . . . . . . .> end as demand_order_ref2
. . . . . . . . . . . . . . . . . . . . . . .> from
dlk_scf_rn_customer_order_li ^Mne
. . . . . . . . . . . . . . . . . . . . . . .> where instr(co_cd,"I")=1;
Heart beat
<<< Invocation of Beeline command completed <<<
Hadoop Job IDs executed by Beeline:
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main],
exit code [2]
Oozie Launcher failed, finishing Hadoop job gracefully
{noformat}
This error was present with Hadoop 2.7.2 and is still visible with Hadoop 2.7.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
