Hi Alex, Typically I have found the first issue is the response time of the database queries. Threads and sockets will start to reach a limit when this happens. Then after that is looked at then the next issue will be understanding what is really possible with the system. Just going from EJB container to Tomcat on an I7 will take at least 5-6msec. Depending on EJB types caching at that layer and the Tomcat layer may be needed. Sorry you are just starting tuning. It took me years to get to sub-millisecond response from Database to web service interface and to see it in my logs below a millisecond and then I started to see latency from network external to my company. Good Hunting. -Tony
--- On Sat, 11/24/12, Alex Moskvin <moskvin.alek...@gmail.com> wrote: From: Alex Moskvin <moskvin.alek...@gmail.com> Subject: Tomcat performance problem To: users@tomcat.apache.org, users-h...@tomcat.apache.org Date: Saturday, November 24, 2012, 5:50 PM Hi, I am using Tomcat 7.0.30 and OpenEJB 4.5 to host high loaded app and at some point found with VisualVM there is a bottleneck when there is a lot of concurrent requests (about 400-600 req/sec). Requests and responses are small (usually not larger than 2kb in JSON format) - almost all CPU time is occupied by TaskQueue.poll() calls and general response latency becomes very slow. I tried APR/Nio connectors and different threading setting and that didn't help anyhow. Tuning (increasing) amount of threads in thread pools didn't help as well. Tests were conducted on Intel Core i7 3450, 8Gb RAM, 100Mb network, RHEL 6.3 x64. How to interpret this relatively low request rate? What I might be doing wrong? What can be tuned in Tomcat to improve request handling? Because we expect much larger request rate processing speed on the mentioned hardware. Here is a VisualVM top output: Hot spots - method; self time [%]; self time; self time (cpu) org.apache.tomcat.util.threads.TaskQueue.poll() 75.508965 57 668 249 ms (75,5%) 0,000 ms net.spy.memcached.internal.OperationFuture.get() 14.22508 10 864 079 ms (14,2%) 6 798 ms net.spy.memcached.internal.BulkGetFuture.internalGet() 5.6411114 4 308 270 ms (5,6%) 0,000 ms org.apache.tomcat.util.net.NioEndpoint$Poller.run() 0.71728086 547 806 ms (0,7%) 530 252 ms org.quartz.core.QuartzSchedulerThread.run() 0.69727546 532 528 ms (0,7%) 0,000 ms Regards, Alex
