On 16/10/2020 10:05, Eric Robinson wrote: > Hi Mark -- > > Those are great questions. See answers below. > > >> -----Original Message----- >> From: Mark Thomas <ma...@apache.org> >> Sent: Friday, October 16, 2020 2:20 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 16/10/2020 00:27, Eric Robinson wrote: >> >> <snip/> >> >>> The localhost_access log shows a request received and an HTTP 200 >> response sent, as follows... >>> >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10- >> 15&multiResFacFi >>> >> lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappproc >> e >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063 HTTP/1.0 >>> ?gizmoid=64438&weekday=5&aptdate=2020-10- >> 15&multiResFacFilterId=0&sess >>> >> ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696& >> rn >>> d2=0.0715816×tamp=15102020125245.789063 200 >>> >>> But WireShark shows what really happened. The server received the GET >> request, and then it sent a FIN to terminate the connection. So if tomcat >> sent >> an HTTP response, it did not make it out the Ethernet card. >>> >>> Is this the weirdest thing or what? Ideas would sure be appreciated! >> >> I am assuming there is a typo in your Java version and you are using Java 8. >> > > Yes, Java 8. > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in less than 6 >> months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) >> then >> you might want to start thinking about it. >> > > Vendor constraint. It's a canned application published by a national software > company, and they have not officially approved tomcat 8 for use on Linux yet. > >> I have a few ideas about what might be going on but rather than fire out >> random theories I have some questions that might help narrow things down. >> >> 1. If this request was successful, how big is the response? >> > > 1035 bytes. > >> 2. If this request was successful, how long would it typically take to >> complete? >> > > Under 60 ms. > >> 3. Looking at the Wireshark trace for a failed request, how long after the >> last >> byte of the request is sent by the client does Tomcat send the FIN? >> > > Maybe 100 microseconds. > >> 4. Looking at the Wireshark trace for a failed request, is the request fully >> sent >> (including terminating CRLF etc)? >> > > Yes, the request as seen by the tomcat server is complete and is terminated > by 0D 0A. > >> 5. Are there any proxies, firewalls etc between the user agent and Tomcat? >> > > User agent -> firewall -> nginx plus -> upstream tomcat servers > >> 6. What timeouts are configured for the Connector? >> > > Sorry, which connector are you referring to? > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? >> > > HTTP/1.1 > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere >> else? > > On the nginx proxy and both upstream tomcat servers. (On the user agent, too, > but that doesn't help us in this case.) > > If you would like to see a screen shot showing all 4 captures side-by-size, I > can send you a secure link. It will verify my answers above. It shows 4 > separate WireShark captures taken simultaneously: > > (a) the request going from the nginx proxy to tomcat 1 > (b) tomcat 1 receiving the request and terminating the connection > (c) nginx sending the request to tomcat 2 > (d) tomcat 2 replying to the request (but the reply does not help the user > because the tomcat server does not recognize the user agent's JSESSIONID > cookie, so it responds "invalid session."
Hmm. That rules out most of my ideas. I'd like to see those screen shots please. Better still would be access to the captures themselves (just the relevant connections not the whole thing). I believe what you are telling us but long experience tells me it is best to double check the original data as well. I have observed something similar ish in the CI systems. In that case it is the requests that disappear. Client side logging shows the request was made but there is no sign of it ever being received by Tomcat. I don't have network traces for that (yet) so I'm not sure where the data is going missing. I am beginning to suspect there is a hard to trigger Tomcat or JVM bug here. I think a Tomcat bug is more likely although I have been over the code several times and I don't see anything. A few more questions: Which HTTP connector are you using? BIO, NIO or APR/Native? Is the issue reproducible if you switch to a different connector? How easy is it for you to reproduce this issue? How are you linking the request you see in the access log with the request you see in Wireshark? How comfortable are you running a patched version of Tomcat (drop class files provided by me into $CATALINA_BASE/lib in the right directory structure and restart Tomcat)? Just thinking ahead about collecting additional debug information. Thanks, Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
