When testing our WebSocket application under heavy load we eventually get the
following exception from the JSR-356 ServerEndpoint's RemoteEndpoint.Basic when
trying to write our binary response message:
java.lang.IllegalStateException: The remote endpoint was in state
[BINARY_FULL_WRITING] which is an invalid state for called method
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.checkState(WsRemoteEndpointImplBase.java:1048)
~[tomcat7-websocket.jar:7.0.48-dev]
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.binaryStart(WsRemoteEndpointImplBase.java:1001)
~[tomcat7-websocket.jar:7.0.48-dev]
at
org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemoteEndpointImplBase.java:118)
~[tomcat7-websocket.jar:7.0.48-dev]
at
org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemoteEndpointBasic.java:43)
~[tomcat7-websocket.jar:7.0.48-dev]
at
com.thingworx.communications.server.connection.Jsr356ServerConnection.sendBinaryMessage(Jsr356ServerConnection.java:240)
~[thingworx-server-common.jar:na]
at
com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendBinaryMessage(CommunicationEndpoint.java:321)
~[thingworx-server-common.jar:na]
at
com.thingworx.communications.common.endpoints.CommunicationEndpoint.sendMessage(CommunicationEndpoint.java:303)
~[thingworx-server-common.jar:na]
at
com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(RoutingServerEndpoint.java:24)
~[thingworx-apiserver.jar:na]
at
com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(OutboundMessageRoutingTask.java:79)
~[thingworx-apiserver.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on Ubuntu
Server 13.10 in EC2 (16 vCPUs). The client created 1000 concurrent websockets,
and is using 100 threads to send 100 concurrent requests every 3 ms. The
server receives the messages, processes them, then sends the response back to
the appropriate client. Many response are sent successfully, but after some
period of time, we start receiving the above errors when attempting to send the
responses back.
Is it possible that we could be filling up the TCP buffers (or something
related) and this would cause the above exception?
Please note: we lock around a specific ServerEndpoint's RemoteEndpoint.Basic -
to make sure we aren't trying to send more than 1 message at a time.
Please let me know if this is a bug, or if you need additional information.
Thanks,
Bob
