Attention Tomcat developers who know how the Coyote bit of Tomcat works… I’ve got a tricky one for you! :-)
We’re struggling with a puzzling problem where intermittently, calls to a servlet are delivered back to the client with all of the headers missing except the ones that Coyote adds. In roughly 9,999 out of 10,000 attempts the response returns correctly (which includes a number of headers which are added by filters, a body response which is gzipped by the servlet and a “Content-Encoding” header which indicates it’s gzipped, and the correct “Content-Length” header). But, intermittently, approximately 1 time out of 10,000 requests, the response back to the client contains only 3 headers: “Date”, “Server”, and a “Transfer-Encoding” header set to “chunked” (even though the response body is not chunked). The result of this is that the gzipped body is presented back to the client with the “Content-Encoding” header missing so it’s rendered unreadable by the browser, and all of the other headers are missing. (We’ve determined this by being able to capture a couple of instances of this issue in Fiddler). Looking at the Coyote source code it would appear that if the “Content-Length” header isn’t presented to Coyote from the upstream handling of the filters and servlet then it’ll default to a chunked encoding, which explains why the response is getting chunked! So, basically - the filters run without error, the servlet runs without error, but the when the response is received back into Coyote’s AbstractHttp11Processor.process() method, the headers have disappeared! Annoyingly we can’t find a recreatable test case, but anecdotally it’s when the server is heavily loaded, but there are still plenty of pooled workers available, and plenty of memory. This is a very high volume server (probably millions of requests a day) so adding verbose logging within Coyote isn’t a realistic option. I don’t believe it’s actually possible for the filters or the servlet to remove headers even if they want to (they don’t appear to have any access to the MimeHeaders of the Coyote “Response” object, and the HttpServletResponse object doesn’t have any methods which allow header removal). The only clue we have is occasionally we see “INFO: Encountered a non-recycled response and recycled it forcedly.” in the logs, but these errors are infrequent compared to the number of instances of the problem, and the timestamps of these errors don’t correspond to when users have reported the error. My best guess would be somehow the Coyote “Request” object is accidentally accessed by two threads concurrently to service two requests, and one request is calling the Request.recycle() (which is calling headers.recycle() which removes all of the headers) but I’m struggling to see how that could happen. Note we are not using async servlets. I appreciate 7.0.72 isn’t the very latest and greatest version but I can’t find anything in the change log for later 7.x, 8.x, 8.5.x or 9.x releases which indicate this is a known issue or resolved. Any ideas about how to further diagnose this would be very gratefully received. Otherwise our next step is to blindly update to later versions and just hope it goes away. Thanks, Richard
