>
> In each of the four thread dumps, there are (the same) 4 threads
> connected to and waiting for an answer from a PHP FCGI backend behind
> Tomcat. You need to investigate, why that backend isn't sending a
> response (or takes so long for it).
>
> Regards,
>
> Rainer
>
Hi Rainer,
I have debugged PHP, the process ends normally and the response is
sent 100% of the time, so I don't know why the tomcat thread keeps
listening to it.
There are always a maximum of 4 threads that keeps reading from the
PHP Input Stream. As long as there are 4 "hanging" threads all other
requests to tomcat work perfectly, which is very strange.
The thread dumps always show this:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39)
...
At some point (up to 24 hours) tomcat "cleans" the threads and they
stop. And after a while it begins with the failures until there are
again 4 "hanging" threads.
If I look in VisualVM in the tab MBeans at the PhpCGIServlet of those
processes, there is a maxTime of 3406, so PHP should not be the
problem.
A heapdump shows nothing unusual I think:
Total Bytes: 35.381.250
Total Classes: 6.655
Total Instances: 454.066
Classloaders: 178
GC Roots: 3.318
Number of Objects Pending for Finalization: 0
I am really out of ideas...
> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design:
> >>> From: Daniel Castilla | thin(k)design [mailto:d...@thin-k-design.com]
> >>> Sent: Thursday, March 14, 2019 12:37 PM
> >>> To: Jäkel, Guido <g.jae...@dnb.de>; users@tomcat.apache.org
> >>> Subject: Re: Crash in http connector random once a day
> >>>
> >>> Dear Guido,
> >>>
> >>> thanks for the reply. The requests are reaching tomcat, and a thread
> >>> is always started, if I look at the current threads on the tomcat
> >>> manager I see the following, there are 4 threads that are processing
> >>> since 2+ hours:
> >>>
> >>> R ? ? ? ? ? ?
> >>> S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost GET
> >>> /manager/status HTTP/1.1
> >>> S 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words&id=17385 HTTP/1.1
> >>> S 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words&id=18986 HTTP/1.1
> >>> S 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> /cloudworx/?method=words&id=10560 HTTP/1.1
> >>> R ? ? ? ? ? ?
> >>> S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 localhost POST
> >>> >/cloudworx/?method=words&id=10560 HTTP/1.1
> >>>
> >>> I'm not sure what other metrics would be helpful, but your Unix script
> >>> wouldn't help much, as I am on a Windows Server 2012 and I would like
> >>> to avoid installing Cygwin or something similar.
> >>
> >> Dear Daniel,
> >>
> >> the script just read-out the same core data and does some pretty print.
> >> You may do it by your own by doing a HTTP-Request against
> >>
> >>
> >> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Catalina:type=RequestProcessor,*'
> >>
> >>
> >> What is interesting from your snippet that there are POST Request "in
> >> Service" (i.e. Progress) since more than 2h. And you told that the ID is
> >> unique, but there are two times a '10560'.
> >>
> >> Is there a database service involved in the backend?
> >>
> >> BTW: In addition to pull static thread dumps you may use JVisualVM to get
> >> a live view to the threads.
> >
> > Dear Guido and Mark,
> >
> > the same Id '10560' appears twice in the URL as the same request was
> > retried for two times, the third time it worked.
> > The other two URLs had only one retry, the second time it worked.
> >
> > You can find a complete thread dump here (I hope the link gets
> > through):
> > https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/view?usp=sharing
> >
> > I had to restart tomcat now in order to access the jmxproxy as it
> > wasn't active in the user roles, so the active "problem" processes are
> > gone for now. But it will happen again, at latest tomorrow.
> >
> > I tried JVisualVM, but I am no Java Expert and don't know what to search
> > for...
> >
> > Thanks
> > Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
