> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, November 13, 2020 5:47 PM
> To: users@tomcat.apache.org
> Subject: Re: Weirdest Tomcat Behavior Ever?
>
> Eric sent me a copy of the strace (thanks Eric) and while it is consistent
> with
> what has already been observed, it didn't provide any new information on
> the socket / file descriptor being closed.
>
> I'd like to suggest running again with the following:
>
> sudo strace -r -f -e trace=network,desc -p <tomcat pid>
>
> That should log the file descriptor being closed (and other fd activity).
> There
> are a couple of things we might be able to do with this:
>
> - we'll be able to determine if the socket is closed on the same or a
> different thread
> - we might be able to correlate the time of closure with other logs
> (seems unlikely as we have this from Wireshark but you never know)
> - the class before the close might be enlightening
>
> Mark
We will do that first thing Monday when the users start work. Hopefully it will
turn up something!
-Eric
>
> On 13/11/2020 22:05, Paul Carter-Brown wrote:
> > lol, and there I was feeling ignored :-)
> >
> > That was the first thing I would have looked at. Is the OS reporting
> > errors to the JVM writing data or is the JVM not writing the data.
> > Strace will tell you this quite easily.
> >
> >
> > On Fri, Nov 13, 2020 at 5:27 PM Eric Robinson
> > <eric.robin...@psmnv.com>
> > wrote:
> >
> >>
> >>> -----Original Message-----
> >>> From: Paul Carter-Brown <p...@ukheshe.co.za>
> >>> Sent: Friday, October 16, 2020 6:11 AM
> >>> To: Tomcat Users List <users@tomcat.apache.org>
> >>> Subject: Re: Weirdest Tomcat Behavior Ever?
> >>>
> >>> Hi Eric,
> >>>
> >>> These weird situations are sometimes best looked at by confirming
> >>> what
> >> the
> >>> OS is seeing from user-space.
> >>>
> >>> Can you run: sudo strace -r -f -e trace=network -p <tomcat pid>
> >>>
> >>> You can then log that to a file and correlate and see if the kernel
> >>> is
> >> in fact
> >>> being asked to send the response.
> >>>
> >>> It's very insightful to see what is actually going on between the
> >>> JVM
> >> and
> >>> Kernel.
> >>>
> >>> Paul
> >>
> >> Paul, this message went to spam and I just found it!
> >>
> >> I will try this suggestion immediately.
> >>
> >> -Eric
> >>
> >>>
> >>> On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas <ma...@apache.org>
> wrote:
> >>>
> >>>> On 16/10/2020 10:05, Eric Robinson wrote:
> >>>>> Hi Mark --
> >>>>>
> >>>>> Those are great questions. See answers below.
> >>>>>
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Mark Thomas <ma...@apache.org>
> >>>>>> Sent: Friday, October 16, 2020 2:20 AM
> >>>>>> To: users@tomcat.apache.org
> >>>>>> Subject: Re: Weirdest Tomcat Behavior Ever?
> >>>>>>
> >>>>>> On 16/10/2020 00:27, Eric Robinson wrote:
> >>>>>>
> >>>>>> <snip/>
> >>>>>>
> >>>>>>> The localhost_access log shows a request received and an HTTP
> >>>>>>> 200
> >>>>>> response sent, as follows...
> >>>>>>>
> >>>>>>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET
> >>>>>>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10-
> >>>>>> 15&multiResFacFi
> >>>>>>>
> >>>>>>
> >>>
> lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztapppr
> >>>>>> oc
> >>>>>> e
> >>>>>>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063
> >>> HTTP/1.0
> >>>>>>> ?gizmoid=64438&weekday=5&aptdate=2020-10-
> >>>>>> 15&multiResFacFilterId=0&sess
> >>>>>>>
> >>>>>>
> >>>
> ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696
> >>>>>> &
> >>>>>> rn
> >>>>>>> d2=0.0715816×tamp=15102020125245.789063 200
> >>>>>>>
> >>>>>>> But WireShark shows what really happened. The server received
> >>>>>>> the GET
> >>>>>> request, and then it sent a FIN to terminate the connection. So
> >>>>>> if
> >>>> tomcat sent
> >>>>>> an HTTP response, it did not make it out the Ethernet card.
> >>>>>>>
> >>>>>>> Is this the weirdest thing or what? Ideas would sure be
> >> appreciated!
> >>>>>>
> >>>>>> I am assuming there is a typo in your Java version and you are
> >>>>>> using
> >>>> Java 8.
> >>>>>>
> >>>>>
> >>>>> Yes, Java 8.
> >>>>>
> >>>>>> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in
> >>>>>> less
> >>>> than 6
> >>>>>> months). If you aren't already planning to upgrade (I'd suggest
> >>>>>> to
> >>>> 9.0.x) then
> >>>>>> you might want to start thinking about it.
> >>>>>>
> >>>>>
> >>>>> Vendor constraint. It's a canned application published by a
> >>>>> national
> >>>> software company, and they have not officially approved tomcat 8
> >>>> for use on Linux yet.
> >>>>>
> >>>>>> I have a few ideas about what might be going on but rather than
> >>>>>> fire out random theories I have some questions that might help
> >>>>>> narrow things
> >>>> down.
> >>>>>>
> >>>>>> 1. If this request was successful, how big is the response?
> >>>>>>
> >>>>>
> >>>>> 1035 bytes.
> >>>>>
> >>>>>> 2. If this request was successful, how long would it typically
> >>>>>> take to complete?
> >>>>>>
> >>>>>
> >>>>> Under 60 ms.
> >>>>>
> >>>>>> 3. Looking at the Wireshark trace for a failed request, how long
> >>>>>> after
> >>>> the last
> >>>>>> byte of the request is sent by the client does Tomcat send the FIN?
> >>>>>>
> >>>>>
> >>>>> Maybe 100 microseconds.
> >>>>>
> >>>>>> 4. Looking at the Wireshark trace for a failed request, is the
> >>>>>> request
> >>>> fully sent
> >>>>>> (including terminating CRLF etc)?
> >>>>>>
> >>>>>
> >>>>> Yes, the request as seen by the tomcat server is complete and is
> >>>> terminated by 0D 0A.
> >>>>>
> >>>>>> 5. Are there any proxies, firewalls etc between the user agent
> >>>>>> and
> >>>> Tomcat?
> >>>>>>
> >>>>>
> >>>>> User agent -> firewall -> nginx plus -> upstream tomcat servers
> >>>>>
> >>>>>> 6. What timeouts are configured for the Connector?
> >>>>>>
> >>>>>
> >>>>> Sorry, which connector are you referring to?
> >>>>>
> >>>>>> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS?
> >>>>>>
> >>>>>
> >>>>> HTTP/1.1
> >>>>>
> >>>>>> 8. Where are you running Wireshark? User agent? Tomcat?
> Somewhere
> >>>>>> else?
> >>>>>
> >>>>> On the nginx proxy and both upstream tomcat servers. (On the user
> >>>>> agent,
> >>>> too, but that doesn't help us in this case.)
> >>>>>
> >>>>> If you would like to see a screen shot showing all 4 captures
> >>>> side-by-size, I can send you a secure link. It will verify my
> >>>> answers above. It shows 4 separate WireShark captures taken
> simultaneously:
> >>>>>
> >>>>> (a) the request going from the nginx proxy to tomcat 1
> >>>>> (b) tomcat 1 receiving the request and terminating the connection
> >>>>> (c) nginx sending the request to tomcat 2
> >>>>> (d) tomcat 2 replying to the request (but the reply does not help
> >>>>> the
> >>>> user because the tomcat server does not recognize the user agent's
> >>>> JSESSIONID cookie, so it responds "invalid session."
> >>>>
> >>>> Hmm.
> >>>>
> >>>> That rules out most of my ideas.
> >>>>
> >>>> I'd like to see those screen shots please. Better still would be
> >>>> access to the captures themselves (just the relevant connections
> >>>> not the whole thing). I believe what you are telling us but long
> >>>> experience tells me it is best to double check the original data as
> >> well.
> >>>>
> >>>> I have observed something similar ish in the CI systems. In that
> >>>> case it is the requests that disappear. Client side logging shows
> >>>> the request was made but there is no sign of it ever being received
> >>>> by Tomcat. I don't have network traces for that (yet) so I'm not
> >>>> sure where the data is going missing.
> >>>>
> >>>> I am beginning to suspect there is a hard to trigger Tomcat or JVM
> >>>> bug here. I think a Tomcat bug is more likely although I have been
> >>>> over the code several times and I don't see anything.
> >>>>
> >>>> A few more questions:
> >>>>
> >>>> Which HTTP connector are you using? BIO, NIO or APR/Native?
> >>>>
> >>>> Is the issue reproducible if you switch to a different connector?
> >>>>
> >>>> How easy is it for you to reproduce this issue?
> >>>>
> >>>> How are you linking the request you see in the access log with the
> >>>> request you see in Wireshark?
> >>>>
> >>>> How comfortable are you running a patched version of Tomcat (drop
> >>>> class files provided by me into $CATALINA_BASE/lib in the right
> >>>> directory structure and restart Tomcat)? Just thinking ahead about
> >>>> collecting additional debug information.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Mark
> >>>>
> >>>> -------------------------------------------------------------------
> >>>> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>
> >>>>
> >> Disclaimer : This email and any files transmitted with it are
> >> confidential and intended solely for intended recipients. If you are
> >> not the named addressee you should not disseminate, distribute, copy or
> alter this email.
> >> Any views or opinions presented in this email are solely those of the
> >> author and might not represent those of Physician Select Management.
> >> Warning: Although Physician Select Management has taken reasonable
> >> precautions to ensure no viruses are present in this email, the
> >> company cannot accept responsibility for any loss or damage arising
> >> from the use of this email or attachments.
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
