Ricardo, That exception gets raised by the DefaultNodeManager when all RiakNode instances are reporting that no connections are available from the pool - e.g. a max number of connections have been defined and they're all out of the pool and in use.
Not sure why that would be happening; if connections get closed by the remote peer the socket is discarded and the permit is returned to the pool's controlling semaphore. Could you enable logging in the client (set to debug)? The RiakNode will spit out a ton of info that should let us see what's going on with the connections. (The client uses SLF4J so you just need to configure a logger in the dependencies) Thanks, - Roach On Wed, Feb 25, 2015 at 2:54 PM, Ricardo Mayerhofer <ricardo....@gmail.com> wrote: > Hi all, > We're deploying a new application using Riak to store user cart during > purchase flow. > > The application runs fine, however after a few hours all Riak operation > fails on the client side, even if the cluster is up and running ok. > > The full stack exception is pasted at the end of this e-mail > (com.basho.riak.client.core.NoNodesAvailableException) > > If the application is restarted it gets back working. > > We're using Riak Client 2.0 along with Riak 1.4.10. We're using protocol > buffer with a TPC Load Balancer in front of Riak Cluster. > > The load balancer has a Idle Period Time, so after that time it closes > connection (60 seconds). > > It seem some sort of connection leak. > > Any help is appreciated. Thanks > > <25-02-2015 19:12:09> <Thread : http-nio-8080-exec-150> > <tid:CKT-66147652-c34e-44f8-858a-cc67bf183293> <customerid: > <[ERROR] > [com.b2winc.cart.riak.ShoppingCartRiakRepository] [Error] > ---Stack : com.basho.riak.client.core.NoNodesAvailableException : > java.util.concurrent.ExecutionException at > com.basho.riak.client.core.FutureOperation.get(FutureOperation.java:260) > at > com.basho.riak.client.api.commands.CoreFutureAdapter.get(CoreFutureAdapter.java:52) > at com.basho.riak.client.api.RiakCommand.execute(RiakCommand.java:89) > at com.basho.riak.client.api.RiakClient.execute(RiakClient.java:293) > at > com.b2winc.cart.riak.ShoppingCartRiakRepository.isDependencyWorking(ShoppingCartRiakRepository.java:123) > at > com.b2winc.cart.health.HealthService.getDependencies(HealthService.java:21) > at > com.b2winc.cart.controller.HealthController.health(HealthController.java:24) > at sun.reflect.GeneratedMethodAccessor474.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:618) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at com.ocpsoft.pretty.PrettyFilter.doFilter(PrettyFilter.java:145) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > com.b2winc.checkout.web.ServerErrorFilter.doFilter(ServerErrorFilter.java:24) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > com.b2winc.checkout.web.TransactionIdFilter.doFilter(TransactionIdFilter.java:27) > at > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344) > at > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) > at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676) > at > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:534) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > ---Stack : : com.basho.riak.client.core.NoNodesAvailableException at > com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:95) > at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:197) > at > com.basho.riak.client.core.RiakCluster.retryOperation(RiakCluster.java:328) > at com.basho.riak.client.core.RiakCluster.access$800(RiakCluster.java:44) > at > com.basho.riak.client.core.RiakCluster$RetryTask.run(RiakCluster.java:340) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > We're using > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
