On 16/11/2023 22:53, Pavan Veginati wrote:
Hi,
We are seeing random 404 and 505 errors with GET and POST requests.
Out of the 10 million daily requests in one cluster, there are 2-3 such 404
errors. In another cluster with around 100 million daily requests, we are
seeing 20-30 404s on average per day.
Requests with a 505 error are rare and only happen once or twice a week.
The same requests were successful a few milliseconds before and after the
error. The servlet that handles these requests always returns a 200 with an
appropriate error response if it fails, but never a 404. So, it is surprising
to see a 404 response.
Hmm. Tomcat returns a 505 response in a single location. It does this
when it doesn't understand the HTTP version that has been received in
the request line.
Assuming the clients aren't sending malformed HTTP requests that
suggests the input buffer is getting corrupted. That form of request
mix-up is rare but has happened due to both Tomcat bugs and application
bugs in the past.
I don't think there are any fixes since 8.5.88 that relate to this but
it would be worth considering updating to 8.5.96.
Do you use any of:
- WebSocket
- Servlet asynchronous API
- Sendfile
- HTTP/2
I'm just trying to narrow down the areas of code we may need to look at.
I think the 505s offer the best opportunity to track this down as they
only happen in one place and debug logging is available.
We've only had this issue in our test environment (similar to prod) a couple of
times, but we can't reproduce it again.
I've found a similar issue reported here:
https://lists.apache.org/thread/rxypsggr36joq8fcd44khjky9w4qf4xx
Deployment Details:
Tomcat Version:
8.5.88. Built from source.
Java Version:
JDK 17.
Network Topology:
Direct server to server internal calls with no load balancers in
between.
Are there any known issues that could be causing this,
No.
and are there any specific logs we can enable that might help us debug this
issue further ?
Enable debug logging for the Http11Processor. Add this to
logging.properties:
org.apache.coyote.http11.Http11Processor.level=FINE
That will log additional detail for most failed requests (but not 404s).
Add %r in the access log pattern which will log the entire request line
for every request.
For each 404, check the access log and see whether the request line is
as expected.
For each 505, check both the access log and the standard log files.
If you can provide us with the request lines from the access log for
each 404 (along with whether or not they are as you expect) and the
request line and debug log entry for each 505 we will hopefully be able
draw some further conclusions.
Given you are building from source, one possible investigative option is
providing you with a custom patch that adds additional debug logging. If
we reach the point where that would be helpful, is that something you'd
consider?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
