Hi Juntaek,
I am running Storm 1.0 on AWS .Figured that i have this issue when i have
multiple zookeeper servers. I was using 3 zookeeper servers.
When i make to only one zookeeper there is no issue found . Nimbus and UI
works fine.
Any reason for the nimbus failure when i have multiple zookeepers . Was
seeing below logs in nimbus when i had 3 zookeepers.
Do you have any recommendations on setting zookeeper cluster on ec2
016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report
plugin...
2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for
storm version '1.0.0'
2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: SUSPENDED
2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update:
:disconnected:none
2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: SUSPENDED
2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received
event :disconnected::none: with disconnected Writer Zookeeper.
2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO]
ip-xxxx21-38-235.cloud.nordstrom.net lost leadership.
2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.81.19/xxxx21.81.19:2181, initiating session
2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980003, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to
authenticate using SASL (unknown error)
2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xxxx21.50.34/xxxx21.50.34:2181, initiating session
2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read
additional data from server sessionid 0x156c9158b980002, likely server has
closed socket, closing socket connection and attempting reconnect
2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
to authenticate using SASL (unknown error)
2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session
0x156c9158b980003 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt
to authenticate using SASL (unknown error)
Regards
Pradeep S
On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <[email protected]> wrote:
> We need to see Nimbus log to find out why Nimbus is down. Pasting or
> attaching Nimbus log would be better.
> And please explain your Storm version too.
>
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 26일 (금) 오후 1:16, pradeep s <[email protected]>님이 작성:
>
>> Hi ,
>> While restarting nimbus and ui, i am getting
>> NimbusLeaderNotFoundException..
>> While setting up the cluster also this error came. That time i have
>> pointed zookeper data directory to a new directory and issue was resolved.
>> Any idea on below exception.
>>
>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] none
>> of which is elected as leader, please try again after some time.
>> at
>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85)
>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356)
>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936)
>> at
>> org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93)
>> at
>> org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39)
>> at
>> org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24)
>> at
>> org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106)
>> at clojure.core$some.invoke(core.clj:2570)
>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106)
>> at clojure.lang.RestFn.applyTo(RestFn.java:139)
>> at clojure.core$apply.invoke(core.clj:632)
>> at
>> org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111)
>> at
>> org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56)
>> at
>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>> at
>> org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22)
>> at
>> org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46)
>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230)
>> at
>> org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27)
>> at
>> org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65)
>> at
>> org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55)
>> at
>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103)
>> at
>> org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14)
>> at
>> org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43)
>> at
>> org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160)
>> at
>> org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127)
>> at
>> org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136)
>> at
>> org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown
>> Source)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
>> at
>> org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
>> at
>> org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
