-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Daniel and Rainer,
On 3/15/19 07:56, Daniel Castilla | thin(k)design wrote: >> In each of the four thread dumps, there are (the same) 4 threads >> connected to and waiting for an answer from a PHP FCGI backend >> behind Tomcat. You need to investigate, why that backend isn't >> sending a response (or takes so long for it). >> >> Regards, >> >> Rainer >> > > Hi Rainer, > > I have debugged PHP, the process ends normally and the response is > sent 100% of the time, so I don't know why the tomcat thread keeps > listening to it. > > There are always a maximum of 4 threads that keeps reading from > the PHP Input Stream. As long as there are 4 "hanging" threads all > other requests to tomcat work perfectly, which is very strange. > > The thread dumps always show this: > > java.lang.Thread.State: RUNNABLE at > java.net.SocketInputStream.socketRead0(Native Method) at > java.net.SocketInputStream.read(Unknown Source) at > java.net.SocketInputStream.read(Unknown Source) at > php.java.fastcgi.FCGIInputStream.read(FCGIInputStream.java:39) ... > > At some point (up to 24 hours) tomcat "cleans" the threads and > they stop. And after a while it begins with the failures until > there are again 4 "hanging" threads. > > If I look in VisualVM in the tab MBeans at the PhpCGIServlet of > those processes, there is a maxTime of 3406, so PHP should not be > the problem. It's unclear to me whether the at php.java.servlet.fastcgi.FastCGIServlet.doPost(FastCGIServlet.java:491) method is *handling* a POST -- that is, responding to a POST request to Tomcat or *initiating* a POST to another service. I suspect the answer to the above question will help understand what is going on, here . I find it strange that there is code doing "PHP FastCGI" running within a Java servlet container... what's wrong with regular HTTP? - -chris >> Am 14.03.2019 um 14:11 schrieb Daniel Castilla | thin(k)design: >>>>> From: Daniel Castilla | thin(k)design >>>>> [mailto:d...@thin-k-design.com] Sent: Thursday, March 14, >>>>> 2019 12:37 PM To: Jäkel, Guido <g.jae...@dnb.de>; >>>>> users@tomcat.apache.org Subject: Re: Crash in http >>>>> connector random once a day >>>>> >>>>> Dear Guido, >>>>> >>>>> thanks for the reply. The requests are reaching tomcat, and >>>>> a thread is always started, if I look at the current >>>>> threads on the tomcat manager I see the following, there >>>>> are 4 threads that are processing since 2+ hours: >>>>> >>>>> R ? ? ? ? ? ? S 16 ms 0 KB 0 KB 0:0:0:0:0:0:0:1 >>>>> 0:0:0:0:0:0:0:1 localhost GET /manager/status HTTP/1.1 S >>>>> 7256779 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 >>>>> localhost POST /cloudworx/?method=words&id=17385 HTTP/1.1 S >>>>> 7274046 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 >>>>> localhost POST /cloudworx/?method=words&id=18986 HTTP/1.1 S >>>>> 7228088 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 0:0:0:0:0:0:0:1 >>>>> localhost POST /cloudworx/?method=words&id=10560 HTTP/1.1 R >>>>> ? ? ? ? ? ? S 7290093 ms 0 KB 33 KB 0:0:0:0:0:0:0:1 >>>>> 0:0:0:0:0:0:0:1 localhost POST >>>>> >/cloudworx/?method=words&id=10560 HTTP/1.1 >>>>> >>>>> I'm not sure what other metrics would be helpful, but your >>>>> Unix script wouldn't help much, as I am on a Windows Server >>>>> 2012 and I would like to avoid installing Cygwin or >>>>> something similar. >>>> >>>> Dear Daniel, >>>> >>>> the script just read-out the same core data and does some >>>> pretty print. You may do it by your own by doing a >>>> HTTP-Request against >>>> >>>> URL='http://'${CREDS}'@'${HOST}':'${PORT}'/manager/jmxproxy?qry=Cat alina:type=RequestProcessor,*' >>>> >>>> >>>> >>>> What is interesting from your snippet that there are POST Request "in Service" (i.e. Progress) since more than 2h. And you told that the ID is unique, but there are two times a '10560'. >>>> >>>> Is there a database service involved in the backend? >>>> >>>> BTW: In addition to pull static thread dumps you may use >>>> JVisualVM to get a live view to the threads. >>> >>> Dear Guido and Mark, >>> >>> the same Id '10560' appears twice in the URL as the same >>> request was retried for two times, the third time it worked. >>> The other two URLs had only one retry, the second time it >>> worked. >>> >>> You can find a complete thread dump here (I hope the link gets >>> through): >>> https://drive.google.com/file/d/1kfSoJovb0bPHWxg-fh6zD8hTGJ6STq2Z/vi ew?usp=sharing >>> >>> >>> I had to restart tomcat now in order to access the jmxproxy as it >>> wasn't active in the user roles, so the active "problem" >>> processes are gone for now. But it will happen again, at latest >>> tomorrow. >>> >>> I tried JVisualVM, but I am no Java Expert and don't know what >>> to search for... >>> >>> Thanks Daniel > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyMQH0ACgkQHPApP6U8 pFiryQ/9G4NlkK7uT0J3dFt/BvfCXsklp6JRlU06Sp0TTXL8qrScfRVEN3+aTv26 L3orgjkL61XxZKJG2etzMcaLTDDmugr5MNbnB02uyvz5zfoG7MNyaLKbkSQVUYJa 6KB3F0/Se+ajW8pN+GUNplG/e7oefgpCtTVzlt4aPiDbXQV0zVZq4L5UK9RGdmrG b4OLFjE/eL8WK9bQUAb9V/AaqEXpnVMCS7rS38EwgZN6QTyC/Jv3h0AEaFUy8BuY 7o5xzq1/W1F3yAUUEIxFQ4cjmvZid/qjuMT7AmzPZ3Gb0SeIYxuJ6jF31EIc/4Bf LAIQGcpGzE044jMgJLc9r31pw11l8jbWMNC75lx65YVV5+JLEzNZ97yluENsvPZz CfqCusWsOA7K3W+wRdi9be9bgvImjLEbOSLu5nssjUyrHca5+BsJaG2DIc23WRCC 0EO8I/ld9dJ5L+SB4Aa2eejTNuy8Yl8q8/rniBDiHsjS5MoVCWQ2X448F7fVhoJD mryrrA0ug5XMpQjswozBRUXFfuumL9ZkWgPxrc70B5uSZrhpC8k+0Sf9MxX71g1/ vEervNoaslopdrazs0NDkjtbSsLePoieTpK3I07D4WRF8Cv9UekoolO9X2GGdnHk MhDT1vt0B0+UEjerSrBztOslQNiTHdE1t6/kMcaJeNhsG1AP3nA= =RSm+ -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
