On 27/11/2013 17:28, Bob DeRemer wrote: > When testing our WebSocket application under heavy load we eventually get the > following exception from the JSR-356 ServerEndpoint's RemoteEndpoint.Basic > when trying to write our binary response message: > > java.lang.IllegalStateException: The remote endpoint was in state > [BINARY_FULL_WRITING] which is an invalid state for called method > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.checkState(WsRemoteEndpointImplBase.java:1048) > ~[tomcat7-websocket.jar:7.0.48-dev] > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.binaryStart(WsRemoteEndpointImplBase.java:1001) > ~[tomcat7-websocket.jar:7.0.48-dev] > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemoteEndpointImplBase.java:118) > ~[tomcat7-websocket.jar:7.0.48-dev] > at > org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemoteEndpointBasic.java:43) > ~[tomcat7-websocket.jar:7.0.48-dev] > at > com.thingworx.communications.server.connection.Jsr356ServerConnection.sendBinaryMessage(Jsr356ServerConnection.java:240) > ~[thingworx-server-common.jar:na] > at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendBinaryMessage(CommunicationEndpoint.java:321) > ~[thingworx-server-common.jar:na] > at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendMessage(CommunicationEndpoint.java:303) > ~[thingworx-server-common.jar:na] > at > com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(RoutingServerEndpoint.java:24) > ~[thingworx-apiserver.jar:na] > at > com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(OutboundMessageRoutingTask.java:79) > ~[thingworx-apiserver.jar:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on > Ubuntu Server 13.10 in EC2 (16 vCPUs). The client created 1000 concurrent > websockets, and is using 100 threads to send 100 concurrent requests every 3 > ms. The server receives the messages, processes them, then sends the > response back to the appropriate client. Many response are sent > successfully, but after some period of time, we start receiving the above > errors when attempting to send the responses back. > > Is it possible that we could be filling up the TCP buffers (or something > related) and this would cause the above exception?
What can cause the above exception is if the previous message on that endpoint failed with an IOException. The nature of WebSocket is such that once this happens the only option is to close the connection and start a new one. > Please note: we lock around a specific ServerEndpoint's RemoteEndpoint.Basic > - to make sure we aren't trying to send more than 1 message at a time. > > Please let me know if this is a bug, or if you need additional information. A bug is always possible but at this point it looks like some error handling is missing in the app. Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
