Apologies it starts OK and fails at reducer!
--------------------------------------------------------------------------------
Map 1 RUNNING 1 0 1 0 3 0
Reducer 2 INITED 1 0 0 1 0
0p: application_1452068056412_0006 dag:dag_1452068056412_0006_1
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
16/01/06 15:53:53 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM
for app: application_1452068056412_0006 dag: dag_1452068056412_0006_1 vertex:
Map 1
16/01/06 15:53:53 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
Client (1353540671) connection to rhes564/50.140.197.217:56764 from hduser
sending #152
16/01/06 15:53:53 [IPC Client (1353540671) connection to
rhes564/50.140.197.217:56764 from hduser]: DEBUG ipc.Client: IPC Client
(1353540671) connection to rhes564/50.140.197.217:56764 from hduser got value
#152
16/01/06 15:53:53 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus
took 10msfrom hduser]: DEBUG ipc.Client: IPC Client (1353540671) connection to
rhes564/50.140.197.217:56764 from hduser got value #151
16/01/06 15:53:53 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM
for app: application_1452068056412_0006 dag: dag_1452068056412_0006_1 vertex:
Reducer 2
16/01/06 15:53:53 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC
Client (1353540671) connection to rhes564/50.140.197.217:56764 from hduser
sending #153
16/01/06 15:53:53 [IPC Client (1353540671) connection to
rhes564/50.140.197.217:56764 from hduser]: DEBUG ipc.Client: IPC Client
(1353540671) connection to rhes564/50.140.197.217:56764 from hduser got value
#153
16/01/06 15:53:53 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus
took 2ms
Map 1 FAILED 1 0 0 1 4 0
Reducer 2 KILLED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 13.31 s
--------------------------------------------------------------------------------
16/01/06 15:53:53 [main]: INFO SessionState: Map 1: 0(+0,-4)/1 Reducer 2: 0/1
Status: Failed
16/01/06 15:53:53 [main]: ERROR SessionState: Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1452068056412_0006_1_00,
diagnostics=[Task failed, taskId=task_1452068056412_0006_1_00_000000,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:157)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one
out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this message
shall not be understood as given or endorsed by Peridale Technology Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any
responsibility.
From: Mich Talebzadeh [mailto:[email protected]]
Sent: 06 January 2016 15:03
To: [email protected]
Subject: RE: Hive on TEZ fails starting
Not starting at all!
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one
out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this message
shall not be understood as given or endorsed by Peridale Technology Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any
responsibility.
From: Rajesh Balamohan [mailto:[email protected]]
Sent: 06 January 2016 09:18
To: [email protected] <mailto:[email protected]>
Subject: Re: Hive on TEZ fails starting
Is the job starting and getting stuck in the reducer like you mentioned in the
initial mail? or the job itself is not starting?
~Rajesh.B
On Wed, Jan 6, 2016 at 1:47 PM, Mich Talebzadeh <[email protected]
<mailto:[email protected]> > wrote:
Hi,
Thanks for your help. I downloaded and installed snappy libraries as they were
missing.
Setting Hive execution engine to tez and doing a simple query. Hive is stuck
> set hive.execution.engine=tez;
16/01/06 08:20:22 [main]: DEBUG parse.VariableSubstitution: Substitution is on:
tez
In debug mode I get
16/01/06 08:22:07 [IPC Client (2116259755 <tel:%282116259755> ) connection to
localhost/127.0.0.1:8032 <http://127.0.0.1:8032> from hduser]: DEBUG
ipc.Client: IPC Client (2116259755 <tel:%282116259755> ) connection to
localhost/127.0.0.1:8032 <http://127.0.0.1:8032> from hduser got value #188
16/01/06 08:22:07 [main]: DEBUG ipc.ProtobufRpcEngine: Call:
getApplicationReport took 1ms
16/01/06 08:22:07 [LeaseRenewer:hduser@rhes564:9000]: DEBUG hdfs.LeaseRenewer:
Lease renewer daemon for [] with renew id 1 executed
Which does not progress. Sounds like the same issue as I saw with Spark.
Can I do something like?
set tez.master=yarn-client;
Thanks
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one
out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this message
shall not be understood as given or endorsed by Peridale Technology Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any
responsibility.
From: Rajesh Balamohan [mailto:[email protected]
<mailto:[email protected]> ]
Sent: 05 January 2016 11:46
To: [email protected] <mailto:[email protected]>
Subject: Re: Hive on TEZ fails starting
Try ' beeline --hiveconf
tez.task.launch.env="LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native"
--hiveconf
tez.am.launch.env="LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native"
'. Please check if you have the lib*.so available in the native folder (or
point it to the folder which contains the so files)
~Rajesh.B
On Tue, Jan 5, 2016 at 4:00 PM, Mich Talebzadeh <[email protected]
<mailto:[email protected]> > wrote:
Hi,
I have added the following to the LD_LIBRARY_PATH and JAVA_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native
Trying to use TEZ, I still get the same error
0: jdbc:hive2://rhes564:10010/default> set hive.execution.engine=tez;
No rows affected (0.002 seconds)
0: jdbc:hive2://rhes564:10010/default> use oraclehadoop;
No rows affected (0.019 seconds)
0: jdbc:hive2://rhes564:10010/default> select count(1) from sales;
INFO : Tez session hasn't been created yet. Opening session
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1451986680090_0002)
INFO : Map 1: -/- Reducer 2: 0/1
INFO : Map 1: 0/1 Reducer 2: 0/1
INFO : Map 1: 0(+1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-1)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-2)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-2)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-3)/1 Reducer 2: 0/1
INFO : Map 1: 0(+1,-3)/1 Reducer 2: 0/1
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1451986680090_0002_1_00, diagnostics=[Task failed,
taskId=task_1451986680090_0002_1_00_000000, diagnostics=[TaskAttempt 0 failed,
info=[Error: Failure while running task:java.lang.RuntimeException:
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
Dr Mich Talebzadeh
LinkedIn
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
Sybase ASE 15 Gold Medal Award 2008
A Winning Strategy: Running the most Critical Financial Data on ASE 15
http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15",
ISBN 978-0-9563693-0-7.
co-author "Sybase Transact SQL Guidelines Best Practices", ISBN
978-0-9759693-0-4
Publications due shortly:
Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8
Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one
out shortly
http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this message
shall not be understood as given or endorsed by Peridale Technology Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any
responsibility.
From: Rajesh Balamohan [mailto:[email protected]
<mailto:[email protected]> ]
Sent: 05 January 2016 00:35
To: [email protected] <mailto:[email protected]>
Subject: Re: Hive on TEZ fails starting
By default it should add "LD_LIBRARY_PATH" in the container (ref:
https://github.com/apache/tez/blob/abfc8bfb0a8620d31697a31ad516674a8d3f9f7c/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L358)
. In case your snappy native libs are present elsewhere in the cluster
deployment, you can override using "tez.task.launch.env" and
"tez.am.launch.env" (refer:
https://github.com/apache/tez/blob/abfc8bfb0a8620d31697a31ad516674a8d3f9f7c/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L412)
~Rajesh.B
--
~Rajesh.B
--
~Rajesh.B
