Even though it shouldn't be reaching your limits, limit_req does delay in 1 second increments which sounds like it could be responsible for this. You should see error log entries if this happens (severity warning). Have you tried without the limit_req option? You can also use the nodelay option to avoid the delaying behavior.
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req On Thu, Apr 5, 2018 at 6:45 AM, Peter Booth <peter_bo...@me.com> wrote: > John, > > I think that you need to understand what is happening on your host > throughout the duration of the test. Specifically, what is happening with > the tcp connections. If you run netstat and grep for tcp and do this in a > loop every say five seconds then you’ll see how many connections peak get > created. > If the thing you are testing exists in production then you are lucky. You > can do the same in production and see what it is that you need to replicate. > > You didn’t mention whether you had persistent connections (http keep > alive) configured. This is key to maximizing scalability. You did say that > you were using SSL. If it were me I’d use a load generator that more > closely resembles the behavior of real users on a website. Wrk2, Tsung, > httperf, Gatling are examples of some that do. Using jmeter with zero think > time is a very common anti pattern that doesn’t behave anything like real > users. I think of it as the lazy performance tester pattern. > > Imagine a real web server under heavy load from human beings. You will see > thousands of concurrent connections but fewer concurrent requests in > flight. With the jmeter zero think time model then you are either creating > new connections or reusing them - so either you have a shitload of > connections and your nginx process starts running out of file handles or > you are jamming requests down a single connection- neither of which > resemble reality. > > If you are committed to using jmeter for some reason then use more > instances with real thinktimes. Each instance’s connection wil have a > different source port > > Sent from my iPhone > > > On Apr 4, 2018, at 5:20 PM, John Melom <john.me...@spok.com> wrote: > > > > Hi Maxim, > > > > I've looked at the nstat data and found the following values for > counters: > > > >> nstat -az | grep -I listen > > TcpExtListenOverflows 0 0.0 > > TcpExtListenDrops 0 0.0 > > TcpExtTCPFastOpenListenOverflow 0 0.0 > > > > > > nstat -az | grep -i retra > > TcpRetransSegs 12157 0.0 > > TcpExtTCPLostRetransmit 0 0.0 > > TcpExtTCPFastRetrans 270 0.0 > > TcpExtTCPForwardRetrans 11 0.0 > > TcpExtTCPSlowStartRetrans 0 0.0 > > TcpExtTCPRetransFail 0 0.0 > > TcpExtTCPSynRetrans 25 0.0 > > > > Assuming the above "Listen" counters provide data about the overflow > issue you mention, then there are no overflows on my system. While > retransmissions are happening, it doesn't seem they are related to listen > queue overflows. > > > > > > Am I looking at the correct data items? Is my interpretation of the > data correct? If so, do you have any other ideas I could investigate? > > > > Thanks, > > > > John > > > > -----Original Message----- > > From: nginx [mailto:nginx-boun...@nginx.org] On Behalf Of John Melom > > Sent: Tuesday, March 27, 2018 8:52 AM > > To: nginx@nginx.org > > Subject: RE: Nginx throttling issue? > > > > Maxim, > > > > Thank you for your reply. I will look to see if "netstat -s" detects > any listen queue overflows. > > > > John > > > > > > -----Original Message----- > > From: nginx [mailto:nginx-boun...@nginx.org] On Behalf Of Maxim Dounin > > Sent: Tuesday, March 27, 2018 6:55 AM > > To: nginx@nginx.org > > Subject: Re: Nginx throttling issue? > > > > Hello! > > > >> On Mon, Mar 26, 2018 at 08:21:27PM +0000, John Melom wrote: > >> > >> I am load testing our system using Jmeter as a load generator. > >> We execute a script consisting of an https request executing in a > >> loop. The loop does not contain a think time, since at this point I > >> am not trying to emulate a “real user”. I want to get a quick look at > >> our system capacity. Load on our system is increased by increasing > >> the number of Jmeter threads executing our script. Each Jmeter thread > >> references different data. > >> > >> Our system is in AWS with an ELB fronting Nginx, which serves as a > >> reverse proxy for our Docker Swarm application cluster. > >> > >> At moderate loads, a subset of our https requests start experiencing > >> to a 1 second delay in addition to their normal response time. The > >> delay is not due to resource contention. > >> System utilizations remain low. The response times cluster around 4 > >> values: 0 millilseconds, 50 milliseconds, 1 second, and 1.050 > >> seconds. Right now, I am most interested in understanding and > >> eliminating the 1 second delay that gives the clusters at 1 second and > >> 1.050 seconds. > >> > >> The attachment shows a response time scatterplot from one of our runs. > >> The x-axis is the number of seconds into the run, the y-axis is the > >> response time in milliseconds. The plotted data shows the response > >> time of requests at the time they occurred in the run. > >> > >> If I run the test bypassing the ELB and Nginx, this delay does not > >> occur. > >> If I bypass the ELB, but include Nginx in the request path, the delay > >> returns. > >> > >> This leads me to believe the 1 second delay is coming from Nginx. > > > > There are no magic 1 second delays in nginx - unless you've configured > something explicitly. > > > > Most likely, the 1 second delay is coming from TCP retransmission > timeout during connection establishment due to listen queue overflows. > Check "netstat -s" to see if there are any listen queue overflows on your > hosts. > > > > [...] > > > > -- > > Maxim Dounin > > http://mdounin.ru/ > > _______________________________________________ > > nginx mailing list > > nginx@nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > > > > ________________________________ > > NOTE: This email message and any attachments are for the sole use of the > intended recipient(s) and may contain confidential and/or privileged > information. Any unauthorized review, use, disclosure or distribution is > prohibited. If you have received this e-mail in error, please contact the > sender by replying to this email, and destroy all copies of the original > message and any material included with this email. > > _______________________________________________ > > nginx mailing list > > nginx@nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > > > > ________________________________ > > NOTE: This email message and any attachments are for the sole use of the > intended recipient(s) and may contain confidential and/or privileged > information. Any unauthorized review, use, disclosure or distribution is > prohibited. If you have received this e-mail in error, please contact the > sender by replying to this email, and destroy all copies of the original > message and any material included with this email. > > _______________________________________________ > > nginx mailing list > > nginx@nginx.org > > http://mailman.nginx.org/mailman/listinfo/nginx > _______________________________________________ > nginx mailing list > nginx@nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx >
_______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
