-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Eric,
On 9/9/20 20:42, Eric Robinson wrote: > Hi Chris -- > >> Are you have any specific problem you are trying to diagnose or >> fix? Or are you just academically interested in what conditions >> might cause "slow" >> request processing? > > A little of both. We've been running about 1500 instances of > tomcat for the past 15 years. We're not tomcat experts by any > means, but we're always looking to refine our understanding of > tomcat performance. Like many people, we have custom scripts (ours > are in python) that parse the jasper logs and produce a report > that summarizes responsiveness and helps us isolate underperforming > tomcat instances and JSP calls. Occasionally, we see evidence of > chronic high latency in processing time when there is no indication > of bottlenecks or problems in the servers themselves or the > database back-ends. We theorize that client connectivity could be > responsible. That is a reasonable conclusion. You should also look at worker-thread availability. When you see these "high latency" (which is usually a term reserved for I/O characterization) events, do you have: 1. Available worker threads (from the executor thread pool) 2. Any other shared/limited resource (e.g. DB connection pool) Also, are you seeing the otherwise unexpected slowness on each Tomcat node, or are you seeing it at the load-balancer/multiplexer level? - -chris >> -----Original Message----- From: Christopher Schultz >> <ch...@christopherschultz.net> Sent: Wednesday, September 9, 2020 >> 7:41 AM To: users@tomcat.apache.org Subject: Re: Tomcat >> Processing Timer Question >> > Eric, > > On 9/8/20 17:29, Eric Robinson wrote: >>>> Got it. So TCP retransmits can impact tomcat processing time >>>> under certain conditions, more likely due to issues with >>>> receiving requests from the client than sending responses. > Well... buffering can happen either during the client-write phase > or the server-read phase or both. > > Imagine a slow network like EDGE or something similar where the > first byte arrives at Tomcat's poller and it handed-off to the > request-processor (t=0 as far as Tomcat is concerned) and uploads a > large image over that EDGE connection. The OS won't allocate an > infinite input buffer, so at some point the Poller will get byte 0 > when the client hasn't uploaded the complete request. It may still > take several seconds to upload all those bytes. > > Imagine that the response is a transformed image so the response is > also large. The OS won't allocate an infinite output buffer, so at > some point the bytes will start streaming to the client (at a slow > rate). When the output buffer fills, your request-processing thread > will stall when calling ServletOutputStream.write() to write those > image bytes. > > If your image transform is instantaneous, your access log will > report that the request took "a long time" relative to the amount > of time spent actually processing the request. Basically, you are > just waiting on I/O the entire time. > > Are you have any specific problem you are trying to diagnose or > fix? Or are you just academically interested in what conditions > might cause "slow" request processing? > > Hope that helps, -chris > >>>>> -----Original Message----- From: Mark Thomas >>>>> <ma...@apache.org> Sent: Tuesday, September 8, 2020 4:05 PM >>>>> To: users@tomcat.apache.org Subject: Re: Tomcat Processing >>>>> Timer Question >>>>> >>>>> On 08/09/2020 21:46, Eric Robinson wrote: >>>>>> Hi Mark -- >>>>>> >>>>>> "If the request is split across multiple packets the >>>>>> timer starts when Tomcat >>>>> reads the first byte of the request from the first packet. >>>>>> Tomcat stops the timer on a request after the last byte >>>>>> of the response has >>>>> been accepted by the network stack." >>>>>> >>>>>> Now we're getting somewhere. If tomcat starts its timer >>>>>> when it reads the >>>>> first byte of the client's request, and the request is >>>>> split into multiple packets, doesn't it stand to reason >>>>> that the timer would run longer when there are TCP >>>>> retransmits? >>>>> >>>>> For the request, it depends. If the retransmit is for part >>>>> of the request body and Tomcat hasn't read that far yet (or >>>>> starting reading at all) then it probably won't impact the >>>>> processing time. If Tomcat is performing a read and waiting >>>>> for that packet then it will. >>>>> >>>>> For the response, not unless the response is sfficiently >>>>> big and the retransmit sufficiently earlier in the response >>>>> that the TCP buffers fill and Tomcat is blocked from >>>>> further writes. >>>>> >>>>> Mark >>>>> >>>>> >>>>>> >>>>>> --Eric >>>>>> >>>>>>> -----Original Message----- From: Mark Thomas >>>>>>> <ma...@apache.org> Sent: Tuesday, September 8, 2020 >>>>>>> 3:34 PM To: users@tomcat.apache.org Subject: Re: Tomcat >>>>>>> Processing Timer Question >>>>>>> >>>>>>> On 08/09/2020 21:19, Eric Robinson wrote: >>>>>>>> Hi Mark and Christopher, >>>>>>>> >>>>>>>> For clarification, suppose a client sends and HTTP >>>>>>>> POST request which >>>>>>> is bigger than the PMTU and has to be broken into >>>>>>> multiple packets. It sounds like you're saying that the >>>>>>> request is buffered by the network stack, and the stack >>>>>>> does not send it up to tomcat until the full >>>>> request is received. >>>>>>> That would make sense if every HTTP request is >>>>>>> encapsulated in its own separate TCP connection. Most >>>>>>> of the time, that is not the case. A single connection >>>>>>> is held open and used for multiple HTTP requests. The >>>>>>> network stack has no understanding of anything above >>>>>>> TCP, so it does not know when an HTTP request complete. >>>>>>> It must therefore deliver whatever it has, and it would >>>>>>> be up to tomcat to decide when the HTTP request is >>>>>>> complete, wouldn't it? >>>>>>>> >>>>>>>> If that is the case, tomcat could receive a partial >>>>>>>> HTTP request and >>>>>>> would have to wait for the rest before processing it. >>>>>>> So when does tomcat start its processing timer? >>>>>>> >>>>>>> Tomcat starts the processing timer as soon as Tomcat >>>>>>> processes the first bytes of the request. In practice, >>>>>>> this means the network stack has to deliver the data to >>>>>>> Tomcat, the Poller fires a read event, a thread is >>>>>>> allocated to process that read event, any TLS handshake >>>>>>> has completed and Tomcat has read the first real byte >>>>>>> of the request. >>>>>>> >>>>>>> If the request is split across multiple packets the >>>>>>> timer starts when Tomcat reads the first byte of the >>>>>>> request from the first packet. >>>>>>> >>>>>>> Tomcat stops the timer on a request after the last byte >>>>>>> of the response has been accepted by the network >>>>>>> stack. >>>>>>> >>>>>>> HTH, >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- From: Christopher >>>>>>>>> Schultz <ch...@christopherschultz.net> Sent: >>>>>>>>> Tuesday, September 8, 2020 1:19 PM To: >>>>>>>>> users@tomcat.apache.org Subject: Re: Tomcat >>>>>>>>> Processing Timer Question >>>>>>>>> >>>>>>>> Eric, >>>>>>>> >>>>>>>> On 9/8/20 13:46, Eric Robinson wrote: >>>>>>>>>>> It is my understanding that the AccessLogValve >>>>>>>>>>> %D field records the time from when the last >>>>>>>>>>> byte of the client's request is received to >>>>>>>>>>> when the last byte of the server's response is >>>>>>>>>>> placed on >>>>> the wire. >>>>>>>>>>> Is that correct? If so, would TCP >>>>>>>>>>> retransmissions impact the timer? >>>>>>>> >>>>>>>> I'm not positive, but I believe Tomcat has zero >>>>>>>> visibility into that level of detail. >>>>>>>> >>>>>>>>>>> If there are connectivity issues between the >>>>>>>>>>> client and server, resulting in TCP >>>>>>>>>>> retransmits, could that appear as higher >>>>>>>>>>> response times in the localhost_access logs? >>>>>>>> >>>>>>>> This would only happen if the re-transmissions were >>>>>>>> to cause network buffering in the OS such that the >>>>>>>> stream writes (at the Java level) were to block (and >>>>>>>> therefore "take time" instead of being essentially >>>>>>> instantaneous). >>>>>>>> >>>>>>>> -chris >>>>>>>>> >>>>>>>>> -------------------------------------------------------------- - --- >> >> >>>>>>>>> - --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For >> additional commands, e-mail: users-h...@tomcat.apache.org > > Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9aKOoACgkQHPApP6U8 pFjeSA/+K0/me8yZTGMcVfCvMrPLTlQENqXNGcTsF8ILgMpz3YVpmqSPjPGGg1W0 xdX8U5saN3T7dLqlLLxGUDoa+YYsIqzQnTYccxqIN4RdBtThem0QKZPrwgNYSSm8 ogzpSXhzG1/aQCMPs0RGFN8vdvCX7gAnqtYd5988b4RFgnMBfrA3Z87cc0kxcsJc nr33L/aMpMf6HJIkX11TPTuQrF8TlplQx2y2RoNxRRdEslHNaxJyDsU03C237On8 7AjicqkRNxJuDbZOu6bCq2em1gi7u22syi50B2X5HRWNyK9PaVhST5ccx34Gref5 y84DUh8Gqaae7AGdj7aY1LeE+enKM3gIjsOaqNDk2X4Q+B05VOhxFwSAo79DNZY+ WX6liPTuO6y6tFW0UmtREnxm6turfzRGfTPUVzGEyAMqP9rBxPuxS7KS39VL7N7n /44zHi30R1beJ8I0fFh5+PRKm9eZsbDniLA5as2oL4Vu2as6e1XmjK+GRn4Pb9U9 jdG+ySFsROeNFg2hkyAXpacLp0PyL1RuiUsACIy8vCTP7jRprrK3aAuV6d4W0pbq +llMh3NVpVzkVGKc5Nc2ydRb5OstQajUmr21u1jH3ze6Rk+I3SmJ2TwWYq4UnpTI z9NweMjRR/RzRsJKDGF7HtMMzva5ywNLiUdawNytE5mKT0buobs= =67LH -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
