Thanks Jung. It was a zookeeper configuration issue.myid file configuration was not proper. Resolved after updating the myid file
On Sun, Aug 28, 2016 at 4:27 AM, Jungtaek Lim <[email protected]> wrote: > As you suspected there seems to be ZK issue. > Unfortunately I don't have experience to operate Storm cluster with EC2. > You may want to check each Zookeeper instance is accessible from your Storm > cluster. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2016년 8월 28일 (일) 오후 5:27, pradeep s <[email protected]>님이 작성: > >> Hi Juntaek, >> I am running Storm 1.0 on AWS .Figured that i have this issue when i have >> multiple zookeeper servers. I was using 3 zookeeper servers. >> When i make to only one zookeeper there is no issue found . Nimbus and UI >> works fine. >> Any reason for the nimbus failure when i have multiple zookeepers . Was >> seeing below logs in nimbus when i had 3 zookeepers. >> Do you have any recommendations on setting zookeeper cluster on ec2 >> >> 016-08-26 23:46:47.822 o.a.s.d.common [INFO] Started statistics report >> plugin... >> 2016-08-26 23:46:47.854 o.a.s.d.nimbus [INFO] Starting nimbus server for >> storm version '1.0.0' >> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980003, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:37.520 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980002, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:37.621 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] >> State change: SUSPENDED >> 2016-08-27 00:07:37.622 o.a.s.zookeeper [INFO] Zookeeper state update: >> :disconnected:none >> 2016-08-27 00:07:37.622 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] >> State change: SUSPENDED >> 2016-08-27 00:07:37.623 o.a.s.c.zookeeper-state-factory [WARN] Received >> event :disconnected::none: with disconnected Writer Zookeeper. >> 2016-08-27 00:07:37.623 o.a.s.zookeeper [INFO] ip-xxxx21-38-235.cloud. >> nordstrom.net lost leadership. >> 2016-08-27 00:07:37.803 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to >> authenticate using SASL (unknown error) >> 2016-08-27 00:07:37.804 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection >> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session >> 2016-08-27 00:07:37.805 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980003, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:38.152 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to >> authenticate using SASL (unknown error) >> 2016-08-27 00:07:38.153 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection >> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session >> 2016-08-27 00:07:38.155 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980002, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:38.395 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.81.19/xxxx21.81.19:2181. Will not attempt to >> authenticate using SASL (unknown error) >> 2016-08-27 00:07:38.396 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection >> established to xxxx21.81.19/xxxx21.81.19:2181, initiating session >> 2016-08-27 00:07:38.398 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980003, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.50.34/xxxx21.50.34:2181. Will not attempt to >> authenticate using SASL (unknown error) >> 2016-08-27 00:07:38.979 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection >> established to xxxx21.50.34/xxxx21.50.34:2181, initiating session >> 2016-08-27 00:07:38.980 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >> additional data from server sessionid 0x156c9158b980002, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2016-08-27 00:07:40.299 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not >> attempt to authenticate using SASL (unknown error) >> 2016-08-27 00:07:40.300 o.a.s.s.o.a.z.ClientCnxn [WARN] Session >> 0x156c9158b980003 for server null, unexpected error, closing socket >> connection and attempting reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at sun.nio.ch.SocketChannelImpl.finishConnect( >> SocketChannelImpl.java:717) >> at org.apache.storm.shade.org.apache.zookeeper. >> ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) >> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$ >> SendThread.run(ClientCnxn.java:1081) >> 2016-08-27 00:07:40.713 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >> connection to server xxxx21.106.227/xxxx21.106.227:2181. Will not >> attempt to authenticate using SASL (unknown error) >> >> Regards >> Pradeep S >> >> On Sat, Aug 27, 2016 at 2:37 AM, Jungtaek Lim <[email protected]> wrote: >> >>> We need to see Nimbus log to find out why Nimbus is down. Pasting or >>> attaching Nimbus log would be better. >>> And please explain your Storm version too. >>> >>> Jungtaek Lim (HeartSaVioR) >>> >>> 2016년 8월 26일 (금) 오후 1:16, pradeep s <[email protected]>님이 작성: >>> >>>> Hi , >>>> While restarting nimbus and ui, i am getting >>>> NimbusLeaderNotFoundException.. >>>> While setting up the cluster also this error came. That time i have >>>> pointed zookeper data directory to a new directory and issue was resolved. >>>> Any idea on below exception. >>>> >>>> org.apache.storm.utils.NimbusLeaderNotFoundException: Found nimbuses [] >>>> none of which is elected as leader, please try again after some time. >>>> at >>>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:85) >>>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:356) >>>> at org.apache.storm.ui.core$fn__12108.invoke(core.clj:936) >>>> at >>>> org.apache.storm.shade.compojure.core$make_route$fn__2467.invoke(core.clj:93) >>>> at >>>> org.apache.storm.shade.compojure.core$if_route$fn__2455.invoke(core.clj:39) >>>> at >>>> org.apache.storm.shade.compojure.core$if_method$fn__2448.invoke(core.clj:24) >>>> at >>>> org.apache.storm.shade.compojure.core$routing$fn__2473.invoke(core.clj:106) >>>> at clojure.core$some.invoke(core.clj:2570) >>>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:106) >>>> at clojure.lang.RestFn.applyTo(RestFn.java:139) >>>> at clojure.core$apply.invoke(core.clj:632) >>>> at >>>> org.apache.storm.shade.compojure.core$routes$fn__2477.invoke(core.clj:111) >>>> at >>>> org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__11576.invoke(json.clj:56) >>>> at >>>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103) >>>> at >>>> org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__4286.invoke(reload.clj:22) >>>> at >>>> org.apache.storm.ui.helpers$requests_middleware$fn__3770.invoke(helpers.clj:46) >>>> at org.apache.storm.ui.core$catch_errors$fn__12301.invoke(core.clj:1230) >>>> at >>>> org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__3474.invoke(keyword_params.clj:27) >>>> at >>>> org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__3514.invoke(nested_params.clj:65) >>>> at >>>> org.apache.storm.shade.ring.middleware.params$wrap_params$fn__3445.invoke(params.clj:55) >>>> at >>>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__3543.invoke(multipart_params.clj:103) >>>> at >>>> org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__3729.invoke(flash.clj:14) >>>> at >>>> org.apache.storm.shade.ring.middleware.session$wrap_session$fn__3717.invoke(session.clj:43) >>>> at >>>> org.apache.storm.shade.ring.middleware.cookies$wrap_cookies$fn__3645.invoke(cookies.clj:160) >>>> at >>>> org.apache.storm.shade.ring.util.servlet$make_service_method$fn__3351.invoke(servlet.clj:127) >>>> at >>>> org.apache.storm.shade.ring.util.servlet$servlet$fn__3355.invoke(servlet.clj:136) >>>> at >>>> org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown >>>> Source) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) >>>> at >>>> org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) >>>> at >>>> org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >>>> at >>>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>
