> From: Mark Thomas <ma...@apache.org> > Sent: Thursday, November 19, 2020 4:34 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 18/11/2020 16:28, Mark Thomas wrote: > > On 18/11/2020 15:41, Eric Robinson wrote: > > <snip/> > > >>I tested it and we're now capturing file descriptor operations. > > <snip/> > > > I am very curious as to what we are going to see in these logs. > > Good news. We are a step closer to figuring out what is going on. > > The new strace logs show the file descriptor associated with the socket being > closed. That means that the socket close MUST be coming from within the > JVM. The key lines in the strace output that show this are (I've removed > unrelated events from between these entries): > > 19166 15:24:21.108653 (+ 0.000020) dup2(50, 336 <unfinished ...> > 19166 15:24:21.108722 (+ 0.000017) <... dup2 resumed>) = 336 > 19166 15:24:21.108778 (+ 0.000027) close(336 <unfinished ...> > 19166 15:24:21.109015 (+ 0.000152) <... close resumed>) = 0 > > Has the Tomcat instance on the machine you obtained the last set of logs > from been restarted since you obtained the logs? If not it should be possible > to identify the thread that triggered the socket close as follows: >
Sounds promising! Unfortunately, the tomcats get restarted every night, so I'll have to watch closely for the error to happen today and I will do a thread dump as close as possible to the time of the error. > Using the following command should trigger the generation of a thread > dump to standard out (which may be redirected to a file). > > kill -3 <tomcat-pid> > > Locate the thread dump. > > Each thread lists its native thread id e.g. nid=0x3522 > > Look for the thread that has nid=0x4ade > > What is the first line for that thread? It should look something like this: > > "http-bio-8080-exec-8" #24 daemon prio=5 os_prio=0 cpu=0.17ms > elapsed=20.54s tid=0x00007f6e546c0800 nid=0x374e waiting on condition > [0x00007f6db38f7000] > > > If the Tomcat instance has been restarted the native thread IDs will have > changed. To identify the thread we'll need to repeat the last set of logs and > once a failure is detected, take a thread dump and provide that along with > the logs and I should be able to join the dots. > > Almost there... > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
