Hi,
I'm looking for advice on how to properly synchronize asynchronous
servlets that use the Java servlet 3.0 async APIs.
Especially, I'm trying to avoid having the servlet experience
IllegalStateExceptions when accessing HttpServletRequest and
HttpServletResponse objects that tomcat has recycled.
The theory I'm working with is that:
The servlet is accessing the HttpServletRequest and
HttpServletResponse objects from non-container threads. For example,
we can assume that an asynchronous operation has completed on some
thread and we want to send the response.
At any time, tomcat may decide to "complete" the request and recycle
the HttpRequest and HttpResponse objects. For example, the socket
could be disconnected or timed out. Tomcat notifies the servlet that
this is happening by calling the AsyncListener.onComplete method.
The non-container thread may be accessing the HttpServletRequest and
HttpServletResponse objects while Tomcat is about to recycle the
objects. So it seems that some sort of synchronization is required at
the servlet level to avoid accessing recycled objects.
The strategy I have implemented is to acquire a lock at the servlet
level when processing the AsyncListener.onComplete callback and also
when accessing the HttpServletRequest and HttpServletResponse objects
from non-container threads. That way, we can be sure that the
non-container thread will never access recycled objects.
However, I've noticed that implementing synchronization at the servlet
level introduces deadlocks between application locks and internal
tomcat locks. Here are two stack traces taken from tomcat 9.0.97:
Non-container thread calling
HttpServletResponse.getOutputStream().close(), while holding an
application-level lock:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.coyote.AsyncStateMachine.asyncError(AsyncStateMachine.java:421)
- waiting to lock <0x00000007fc5feec0> (a
org.apache.coyote.AsyncStateMachine)
at
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:121)
at
org.apache.coyote.AbstractProcessor.handleIOException(AbstractProcessor.java:665)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:388)
at org.apache.coyote.Response.action(Response.java:207)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:258)
at
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:176)
Container thread calling AsyncListener.onComplete(), AsyncListener
implementation trying to acquire the application-level lock:
<application code, stuck trying to acquire the lock held by the
non-container thread>
at
org.apache.catalina.core.AsyncListenerWrapper.fireOnComplete(AsyncListenerWrapper.java:39)
at
org.apache.catalina.core.AsyncContextImpl.fireOnComplete(AsyncContextImpl.java:106)
at
org.apache.coyote.AsyncStateMachine.asyncPostProcess(AsyncStateMachine.java:284)
- locked <0x00000007fc5feec0> (a org.apache.coyote.AsyncStateMachine)
at
org.apache.coyote.AbstractProcessor.asyncPostProcess(AbstractProcessor.java:196)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:83)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
at
org.apache.tomcat.util.net.BwSocketProcessor.doRun(BwSocketProcessor.java:24)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
There is a deadlock between the application-level lock and the
AsyncStateMachine object monitor. They are acquired in different order
by two different threads.
What would be the recommendation to avoid this deadlock?
- Don't synchronize at the application/servlet level, and accept that
sometimes an IllegalStateException will be thrown on the non-container
thread? But then, isn't the IllegalStateException a symptom that a
recycled object is being accessed? What if the recycled object is
already being reused for another purpose? Doesn't that risk cross-talk
between requests?
In general, how can we ensure consistency when accessing the
HttpServletRequest and HttpServletResponse from a non-container thread
while running in async mode?
Thanks in advance,
François
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
