-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Chirag,
On 5/23/13 11:11 PM, Chirag Dewan wrote:
> As I said before a lot of time was spent during the
> ResponseFacade.setContentType() method call. That doesn't tell the
> whole story but more or less the high utilization is mainly in the
> tomcat threads.
Sorry, I forgot to mention another important point: the fact that
ResponseFacade.setContentType takes a relatively long time when
accessing a trivial resource is not that important. Instead, what's
important is how much that call takes relative to your typical
transaction time.
Obviously, you want to limit Tomcat's overhead as much as possible.
But saying that Tomcat 7 is unacceptable because it gets you only 70k
req/sec versus Tomcat 6's 120k req/sec is not exactly fair. What you
really ought to do it benchmark your webapp. If your webapp was only
able to handle 100 tps on Tomcat 6, then it's not suddenly only going
to process 50 tps on Tomcat 7.
Also, you've been throwing-around a lot of different numbers so it's
tough to keep them straight. 70k on Tomcat 7 versus 120k on Tomcat 6,
but Tomcat 6 only gives you 70k req/sec with 25-30% CPU utilization.
But also Tomcat 6 with 48k req/sec for each of 3 clients (that's 150k
req/sec overall) and you are calling it 120k req/sec? I'm totally lost
with all the random numbers you are tossing out here without a)
providing any configuration information such as a <Connector> b)
providing any code or c) providing any rich information from a
recognizable load-test.
You have mentioned both JMeter and "HttpClient" (which I think you
discarded because it was giving you bad performance). There's also
ApacheBench which is freely available and comes with Apache httpd. All
of these tools should produce some kind of report.
For example, if I run ab against my Tomcat 7.0.40 instance just
requesting /index.html as fast as possible long enough to get a CPU
usage number in 'top', I get this report:
Benchmarking 127.0.0.1 (be patient)
Server Software: Apache-Coyote/1.1
Server Hostname: 127.0.0.1
Server Port: 8217
Document Path: /webapp/index.html
Document Length: 3460 bytes
Concurrency Level: 10
Time taken for tests: 85.627 seconds
Complete requests: 49046
Failed requests: 140
(Connect: 0, Receive: 70, Length: 70, Exceptions: 0)
Write errors: 0
Total transferred: 181554032 bytes
HTML transferred: 169456960 bytes
Requests per second: 572.79 [#/sec] (mean)
Time per request: 17.458 [ms] (mean)
Time per request: 1.746 [ms] (mean, across all concurrent requests)
Transfer rate: 2070.61 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 42.9 0 1609
Processing: 0 14 326.4 1 8693
Waiting: 0 1 1.2 1 79
Total: 0 16 329.1 1 8693
"top" showed that Tomcat was taking wildly-varying amounts of CPU:
sometimes 50%, sometimes less than 1%, etc. I never saw it go above
50%. I suspect I've hit some kind of I/O limit, somewhere.
Running again with HTTP keepalives enabled, I have a different report:
Benchmarking 127.0.0.1 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests
Server Software: Apache-Coyote/1.1
Server Hostname: 127.0.0.1
Server Port: 8217
Document Path: /webapp/index.html
Document Length: 3460 bytes
Concurrency Level: 10
Time taken for tests: 57.293 seconds
Complete requests: 1000000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 990005
Total transferred: 3711950025 bytes
HTML transferred: 3460000000 bytes
Requests per second: 17454.15 [#/sec] (mean)
Time per request: 0.573 [ms] (mean)
Time per request: 0.057 [ms] (mean, across all concurrent requests)
Transfer rate: 63270.43 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 12
Processing: 0 1 0.6 1 57
Waiting: 0 1 0.6 0 57
Total: 0 1 0.6 1 57
WARNING: The median and mean for the waiting time are not within a
normal deviation
These results are probably not that reliable.
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 1
80% 1
90% 1
95% 1
98% 1
99% 2
100% 57 (longest request)
Top reported that Tomcat was taking somewhere between 550-600% CPU.
(This is a 4-core hyperthreaded CPU so I have 8 logical cores. 'ab'
was taking about 100% CPU so I think 600% CPU means it was roughly
pegging 6 of my logical cores. Roughly 30% system, 60% user CPU usage.).
All tests were done using the Http11NioProtocol connector with the
default-sizes thread pool (200), but that shouldn't matter as I was
only using c=10 in these tests.
I ran again with hprof taking cpu samples every 10ms. I expected this
to slow things down tremendously. Actually, it looks like it reduced
my CPU usage slightly (10% less? I didn't get a good measurement) but
also performed about the same.
Finished 1000000 requests
Server Software: Apache-Coyote/1.1
Server Hostname: 127.0.0.1
Server Port: 8217
Document Path: /cschultz-chadis/index.html
Document Length: 2181 bytes
Concurrency Level: 10
Time taken for tests: 58.208 seconds
Complete requests: 1000000
Failed requests: 822806
(Connect: 0, Receive: 541, Length: 822265, Exceptions: 0)
Write errors: 1484
Non-2xx responses: 989788
Keep-Alive requests: 982187
Total transferred: 2079010371 bytes
HTML transferred: 1903683155 bytes
Requests per second: 17179.68 [#/sec] (mean)
Time per request: 0.582 [ms] (mean)
Time per request: 0.058 [ms] (mean, across all concurrent requests)
Transfer rate: 34879.63 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.8 0 205
Processing: 0 0 15.3 0 15098
Waiting: 0 0 0.5 0 206
Total: 0 0 15.4 0 15136
Percentage of the requests served within a certain time (ms)
50% 0
66% 0
75% 0
80% 0
90% 0
95% 1
98% 2
99% 3
100% 15136 (longest request)
hprof has the following top-10 CPU eaters during my tests:
CPU SAMPLES BEGIN (total = 64549) Fri May 24 11:49:57 2013
rank self accum count trace method
1 59.15% 59.15% 38180 300889 sun.nio.ch.KQueueArrayWrapper.kevent0
2 18.91% 78.06% 12206 301048
sun.nio.ch.ServerSocketChannelImpl.accept0
3 9.66% 87.72% 6236 301063 java.net.PlainSocketImpl.socketAccept
4 1.79% 89.51% 1155 301110 sun.nio.ch.FileDispatcherImpl.write0
5 1.56% 91.07% 1009 301347 java.lang.Class.getClassLoader0
6 0.80% 91.87% 518 301260 java.lang.Object.<init>
7 0.77% 92.65% 500 300513 java.lang.Throwable.getStackTraceElement
8 0.55% 93.20% 355 301341 sun.nio.ch.KQueueArrayWrapper.interrupt
9 0.50% 93.70% 325 301182 sun.nio.ch.IOUtil.drain
10 0.34% 94.04% 218 300955
java.util.TimeZone.getDefaultInAppContext
RequestFacade.setContentType is nowhere to be found.
It looks to me like the socket manipulation is what's taking so much
time for my test. I could run a longer test (this one was maybe 60
seconds long) to get better data, but I'm not the one motivated to
figure out why Tomcat 7 apparently has such hideous performance that
you aren't willing to deploy it.
If you want to come back with complaints about performance, please
provide sufficient data for someone else to understand what in the
world you are talking about.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRn408AAoJEBzwKT+lPKRYoZsP/jETbu9amlwcoMvHyjYxArvV
vhBnlp/P4bhCWP8VWgrVKrNQNygZ2LVzHd+mszquq0p8rfP4Dfg3CHHHMNc4hM5O
DzxCoCTio/00EHHK5YEhQW4C2y/QO13yVlFr8z0E35b+cDLOGsO2IXhSpDnnkfda
vd+x1ZEEYeY3BYiIHpdpPktHOPDFT2ZkXT6FLenNg3Gi7j/KP5kee23B+AGWOevO
4lTI7Z9/4L1zbTbINCw6CxWGbF5MDTqeifH4HDZr8sLprAaZmKomuUROUhfWrtiq
zHY2BKbLzNKYuwis1HSkCc4h/ZX2h0oVOqLaxl7w1qp03jTL3oEEaoSKdad+FyIR
zIvCQIcb07+hafu+hKL3uopvfVvniKduw7CKzTUsrIJEXDzIzcWk22hw6afPwKLu
I0iqyXhTKY3O88sasUxYDRSsABsfnt2Kluy9D2/biNHXpIvJZ+pdOhcrkti8Wq76
mOP7IORN66IwvquWQGkHxJ4TaZWTQIngyzhwJ3Vsl2m2u50rykZXViODTxRMAmJw
ALmfM/jJj5sBLypt14aok/lJfYgiqYSBnYz9jmDsC/Yo0vV0xO8XBgUxLdWBsRt2
TKPteLvUJbHt67VHuR3oGbVB0n1jUJHJnGl8KfVZHf80NlJj0E0giDYfDLCWAVHy
gD+3ehozR8GKvMa+GIOC
=+3Sa
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
