Prabhu Joseph created ZEPPELIN-2822:
---------------------------------------
Summary: Livy Interpreter should have a way to set Livy RSCConf
RPC_CLIENT_HANDSHAKE_TIMEOUT
Key: ZEPPELIN-2822
URL: https://issues.apache.org/jira/browse/ZEPPELIN-2822
Project: Zeppelin
Issue Type: Bug
Components: livy-interpreter
Affects Versions: 0.7.2
Reporter: Prabhu Joseph
Consider a Node which has NN1 (NameNode) and HiveMetaStore is down but we have
HA for both services. Running livy script will create a new session and will
wait for ipc.client.connect.timeout (20s) for each jar upload into hdfs
{code}
17/07/31 13:59:29 INFO ContextLauncher: 17/07/31 13:59:39 INFO Client: Source
and destination file systems are the same. Not copying
hdfs://prabhu/hdp/apps/2.6.1.0-129/spark/spark-hdp-assembly.jar
17/07/31 13:59:49 INFO ContextLauncher: 17/07/31 13:59:49 INFO Client:
Uploading resource
file:/usr/hdp/current/livy-server/rsc-jars/livy-rsc-0.3.0.2.6.1.0-129.jar ->
hdfs://prabhu/user/diasmi/.sparkStaging/application_1501501991083_0001/livy-rsc-0.3.0.2.6.1.0-129.jar
{code}
and 5 seconds (hive.metastore.client.socket.timeout)
{code}
17/07/26 09:09:46 INFO ContextLauncher: 17/07/26 09:09:46 INFO metastore:
Trying to connect to metastore with URI thrift://prabhu01:9083
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 WARN metastore:
Failed to connect to the MetaStore Server...
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 INFO metastore:
Trying to connect to metastore with URI thrift://prabhu02:9083
17/07/26 09:09:51 INFO ContextLauncher: 17/07/26 09:09:51 INFO metastore:
Connected to metastore.
{code}
and finally will fail with timeout with Livy Server Connect Timeout. 90 Seconds
is too low for this case. Zeppelin has to have a way for overriding this
timeout configuration.
RPC_CLIENT_HANDSHAKE_TIMEOUT("server.connect.timeout", "90s")
{code}
17/07/31 14:00:51 ERROR RSCClient: Failed to connect to context.
java.util.concurrent.TimeoutException: Timed out waiting for context to start.
at
com.cloudera.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:133)
at
com.cloudera.livy.rsc.ContextLauncher.access$200(ContextLauncher.java:62)
at com.cloudera.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:121)
at
io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
