Hi Chris -- > Are you have any specific problem you are trying to diagnose or fix? > Or are you just academically interested in what conditions might cause "slow" > request processing?
A little of both. We've been running about 1500 instances of tomcat for the past 15 years. We're not tomcat experts by any means, but we're always looking to refine our understanding of tomcat performance. Like many people, we have custom scripts (ours are in python) that parse the jasper logs and produce a report that summarizes responsiveness and helps us isolate underperforming tomcat instances and JSP calls. Occasionally, we see evidence of chronic high latency in processing time when there is no indication of bottlenecks or problems in the servers themselves or the database back-ends. We theorize that client connectivity could be responsible. > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Wednesday, September 9, 2020 7:41 AM > To: users@tomcat.apache.org > Subject: Re: Tomcat Processing Timer Question > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Eric, > > On 9/8/20 17:29, Eric Robinson wrote: > > Got it. So TCP retransmits can impact tomcat processing time under > > certain conditions, more likely due to issues with receiving requests > > from the client than sending responses. > Well... buffering can happen either during the client-write phase or the > server-read phase or both. > > Imagine a slow network like EDGE or something similar where the first byte > arrives at Tomcat's poller and it handed-off to the request-processor (t=0 as > far as Tomcat is concerned) and uploads a large image over that EDGE > connection. The OS won't allocate an infinite input buffer, so at some point > the Poller will get byte 0 when the client hasn't uploaded the complete > request. It may still take several seconds to upload all those bytes. > > Imagine that the response is a transformed image so the response is also > large. The OS won't allocate an infinite output buffer, so at some point the > bytes will start streaming to the client (at a slow rate). When the output > buffer fills, your request-processing thread will stall when calling > ServletOutputStream.write() to write those image bytes. > > If your image transform is instantaneous, your access log will report that the > request took "a long time" relative to the amount of time spent actually > processing the request. Basically, you are just waiting on I/O the entire > time. > > Are you have any specific problem you are trying to diagnose or fix? > Or are you just academically interested in what conditions might cause "slow" > request processing? > > Hope that helps, > - -chris > > >> -----Original Message----- From: Mark Thomas <ma...@apache.org> > >> Sent: Tuesday, September 8, 2020 4:05 PM To: > >> users@tomcat.apache.org Subject: Re: Tomcat Processing Timer Question > >> > >> On 08/09/2020 21:46, Eric Robinson wrote: > >>> Hi Mark -- > >>> > >>> "If the request is split across multiple packets the timer starts > >>> when Tomcat > >> reads the first byte of the request from the first packet. > >>> Tomcat stops the timer on a request after the last byte of the > >>> response has > >> been accepted by the network stack." > >>> > >>> Now we're getting somewhere. If tomcat starts its timer when it > >>> reads the > >> first byte of the client's request, and the request is split into > >> multiple packets, doesn't it stand to reason that the timer would run > >> longer when there are TCP retransmits? > >> > >> For the request, it depends. If the retransmit is for part of the > >> request body and Tomcat hasn't read that far yet (or starting reading > >> at all) then it probably won't impact the processing time. If Tomcat > >> is performing a read and waiting for that packet then it will. > >> > >> For the response, not unless the response is sfficiently big and the > >> retransmit sufficiently earlier in the response that the TCP buffers > >> fill and Tomcat is blocked from further writes. > >> > >> Mark > >> > >> > >>> > >>> --Eric > >>> > >>>> -----Original Message----- From: Mark Thomas <ma...@apache.org> > >>>> Sent: Tuesday, September 8, 2020 3:34 PM > >>>> To: users@tomcat.apache.org Subject: Re: Tomcat Processing Timer > >>>> Question > >>>> > >>>> On 08/09/2020 21:19, Eric Robinson wrote: > >>>>> Hi Mark and Christopher, > >>>>> > >>>>> For clarification, suppose a client sends and HTTP POST request > >>>>> which > >>>> is bigger than the PMTU and has to be broken into multiple packets. > >>>> It sounds like you're saying that the request is buffered by the > >>>> network stack, and the stack does not send it up to tomcat until > >>>> the full > >> request is received. > >>>> That would make sense if every HTTP request is encapsulated in its > >>>> own separate TCP connection. Most of the time, that is not the > >>>> case. A single connection is held open and used for multiple HTTP > >>>> requests. The network stack has no understanding of anything above > >>>> TCP, so it does not know when an HTTP request complete. It must > >>>> therefore deliver whatever it has, and it would be up to tomcat to > >>>> decide when the HTTP request is complete, wouldn't it? > >>>>> > >>>>> If that is the case, tomcat could receive a partial HTTP request > >>>>> and > >>>> would have to wait for the rest before processing it. So when does > >>>> tomcat start its processing timer? > >>>> > >>>> Tomcat starts the processing timer as soon as Tomcat processes the > >>>> first bytes of the request. In practice, this means the network > >>>> stack has to deliver the data to Tomcat, the Poller fires a read > >>>> event, a thread is allocated to process that read event, any TLS > >>>> handshake has completed and Tomcat has read the first real byte of > >>>> the request. > >>>> > >>>> If the request is split across multiple packets the timer starts > >>>> when Tomcat reads the first byte of the request from the first > >>>> packet. > >>>> > >>>> Tomcat stops the timer on a request after the last byte of the > >>>> response has been accepted by the network stack. > >>>> > >>>> HTH, > >>>> > >>>> Mark > >>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- From: Christopher Schultz > >>>>>> <ch...@christopherschultz.net> Sent: Tuesday, September 8, 2020 > >>>>>> 1:19 PM To: users@tomcat.apache.org Subject: Re: > >>>>>> Tomcat Processing Timer Question > >>>>>> > >>>>> Eric, > >>>>> > >>>>> On 9/8/20 13:46, Eric Robinson wrote: > >>>>>>>> It is my understanding that the AccessLogValve %D field records > >>>>>>>> the time from when the last byte of the client's request is > >>>>>>>> received to when the last byte of the server's response is > >>>>>>>> placed on > >> the wire. > >>>>>>>> Is that correct? If so, would TCP retransmissions impact the > >>>>>>>> timer? > >>>>> > >>>>> I'm not positive, but I believe Tomcat has zero visibility into > >>>>> that level of detail. > >>>>> > >>>>>>>> If there are connectivity issues between the client and server, > >>>>>>>> resulting in TCP retransmits, could that appear as higher > >>>>>>>> response times in the localhost_access logs? > >>>>> > >>>>> This would only happen if the re-transmissions were to cause > >>>>> network buffering in the OS such that the stream writes (at the > >>>>> Java level) were to block (and therefore "take time" instead of > >>>>> being essentially > >>>> instantaneous). > >>>>> > >>>>> -chris > >>>>>> > >>>>>> ----------------------------------------------------------------- > - -- > >>>>>> > >>>>>> > - -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>> For additional commands, e-mail: > >>>>>> users-h...@tomcat.apache.org > >>>>> > >>>>> Disclaimer : This email and any files transmitted with it are > >>>> confidential and intended solely for intended recipients. If you > >>>> are not the named addressee you should not disseminate, distribute, > >>>> copy or alter this email. Any views or opinions presented in this > >>>> email are solely those of the author and might not represent those > >>>> of Physician > >> Select Management. > >>>> Warning: Although Physician Select Management has taken reasonable > >>>> precautions to ensure no viruses are present in this email, the > >>>> company cannot accept responsibility for any loss or damage arising > >>>> from the use of this email or attachments. > >>>>> > >>>>> ------------------------------------------------------------------ > - -- > >>>>> > >>>>> > - - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>> For additional commands, e-mail: > >>>>> users-h...@tomcat.apache.org > >>>>> > >>>> > >>>> > >>>> ------------------------------------------------------------------- > - -- > >>>> > >>>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: > >>>> users-h...@tomcat.apache.org > >>> > >>> Disclaimer : This email and any files transmitted with it are > >>> confidential and > >> intended solely for intended recipients. If you are not the named > >> addressee you should not disseminate, distribute, copy or alter this > >> email. Any views or opinions presented in this email are solely those > >> of the author and might not represent those of Physician Select > >> Management. Warning: Although Physician Select Management has taken > >> reasonable precautions to ensure no viruses are present in this > >> email, the company cannot accept responsibility for any loss or > >> damage arising from the use of this email or attachments. > >>> > >>> -------------------------------------------------------------------- > - - > >>> > >>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>> > >> > >> > >> --------------------------------------------------------------------- > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > Disclaimer : This email and any files transmitted with it are > > confidential and intended solely for intended recipients. If you are > > not the named addressee you should not disseminate, distribute, copy > > or alter this email. Any views or opinions presented in this email are > > solely those of the author and might not represent those of Physician > > Select Management. Warning: Although Physician Select Management has > > taken reasonable precautions to ensure no viruses are present in this > > email, the company cannot accept responsibility for any loss or damage > > arising from the use of this email or attachments. > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9YzWYACgkQHPApP > 6U8 > pFgy9A//ZTrbPoaA1KLSikqNlxvs+ZXxonYWjDZ73vEvQLVj9P9NDV2/7IcUhe9k > xXZrJXBHQ36Hn4x15EDdJ4/puJQnvWY/+PdcYEHVKkExuUWorGKhb5Tst4hnp > mC/ > i0N3uOrzsePgxJeA22+mBSoSOVmw1dMCu9uWh3H6GmYVXjJKqe25lt0NO/+i > 7Pmp > hrcVusvnikDoBRjg74qKOPbCDfB1iaWBvhgVGfxhRu4OSne/DK0omxT1XtD0or > Ca > 2B6ODVioMnyseliPTWARrzKV7kBV1SnbrAcqdbd4IwgluVwKmY4sa6vtdm8JLz > Xn > 3HbMUyE1o6Z1QcVe0n6lHnlib+4bWxgkvY+4vPDMj3xgJJjhYxBbc9oNGhh51M > NG > tTJpoBqY9Ro0NMQ0EiygnaZzu0ja9c47XBpBiDl4896Cub2kd3+n6uXPMtlk8YQ7 > d63r2LS0y1OEFcgsAeIDXo0A3ehM3SFuWq/l6xDTVXctm2eCsNDfAXtwXGLPK > xfq > H5oJMkwYkFdNok7YVbmBl4FEZpcQNzvEaSIljKpGm/MDbRKBeVEHXLjYTwpg > Gofu > gYC1bQWKUBKAQUIUc7Ys3SXYz7fFmFpYp9swACFNPvMz5yOYNJVaHaQ8BlIJ > 9ksT > AI9qx2ubIPhvIZoRO2nTDIx7eS9lY6DDArvW8XcOSgL4QW9vWrg= > =jdM/ > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
