As you suspected there seems to be ZK issue. Unfortunately I don't have experience to operate Storm cluster with EC2. You may want to check each Zookeeper instance is accessible from your Storm cluster.
Thanks, Jungtaek Lim (HeartSaVioR) 2016년 8월 28일 (일) 오후 5:27, pradeep s <[email protected]>님이 작성: > Hi Juntaek, > I am running Storm 1.0 on AWS .Figured that i have this issue when i have > multiple zookeeper servers. I was using 3 zookeeper servers. > When i make to only one zookeeper there is no issue found . Nimbus and UI > works fine. > Any reason for the nimbus failure when i have multiple zookeepers . Was > seeing below logs in nimbus when i had 3 zookeepers. > Do you have any recommendations on setting zookeeper cluster on ec2 > > 016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report > plugin... > 2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for > storm version '1.0.0' > 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980003, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980002, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] > State change: SUSPENDED > 2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update: > :disconnected:none > 2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] > State change: SUSPENDED > 2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received > event :disconnected::none: with disconnected Writer Zookeeper. > 2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO] > ip-xxxx21-38-235.cloud.nordstrom.net lost leadership. > 2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to > authenticate using SASL (unknown error) > 2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection > established to xxxx21.50.34/xxxx21.50.34:2181, initiating session > 2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980003, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to > authenticate using SASL (unknown error) > 2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection > established to xxxx21.81.19/xxxx21.81.19:2181, initiating session > 2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980002, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to > authenticate using SASL (unknown error) > 2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection > established to xxxx21.81.19/xxxx21.81.19:2181, initiating session > 2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980003, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to > authenticate using SASL (unknown error) > 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection > established to xxxx21.50.34/xxxx21.50.34:2181, initiating session > 2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read > additional data from server sessionid 0x156c9158b980002, likely server has > closed socket, closing socket connection and attempting reconnect > 2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt > to authenticate using SASL (unknown error) > 2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session > 0x156c9158b980003 for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) > at > org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) > at > org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) > 2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket > connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not attempt > to authenticate using SASL (unknown error) > > Regards > Pradeep S > > On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <[email protected]> wrote: > >> We need to see Nimbus log to find out why Nimbus is down. Pasting or >> attaching Nimbus log would be better. >> And please explain your Storm version too. >> >> Jungtaek Lim (HeartSaVioR) >> >> 2016년 8월 26일 (금) 오후 1:16, pradeep s <[email protected]>님이 작성: >> >>> Hi , >>> While restarting nimbus and ui, i am getting >>> NimbusLeaderNotFoundException.. >>> While setting up the cluster also this error came. That time i have >>> pointed zookeper data directory to a new directory and issue was resolved. >>> Any idea on below exception. >>> >>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] >>> none of which is elected as leader, please try again after some time. >>> at >>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85) >>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356) >>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936) >>> at >>> org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93) >>> at >>> org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39) >>> at >>> org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24) >>> at >>> org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106) >>> at clojure.core$some.invoke(core.clj:2570) >>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) >>> at clojure.lang.RestFn.applyTo(RestFn.java:139) >>> at clojure.core$apply.invoke(core.clj:632) >>> at >>> org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111) >>> at >>> org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56) >>> at >>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103) >>> at >>> org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22) >>> at >>> org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46) >>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230) >>> at >>> org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27) >>> at >>> org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65) >>> at >>> org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55) >>> at >>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103) >>> at >>> org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14) >>> at >>> org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43) >>> at >>> org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160) >>> at >>> org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127) >>> at >>> org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136) >>> at >>> org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown >>> Source) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) >>> at >>> org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) >>> at >>> org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >>> at >>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >
