On 12/13/2013 8:12 AM, Bob DeRemer wrote:
Hi Mark (et al),
We’re seeing really high CPU utilization with just a 1000 active
websockets that are sending/receiving 100 byte binary messages. The
profiling snippet below was with 1000 connections. If we start to
increase the number of connections, the time spent in the NIO logic
starts to consume the majority of the processing. As a result, I’m
trying to verify:
How many cores does your cpu have? You might be thrashing your cpu with
excessive context switching because of the large number of threads. If
that's the case, then throwing more lower-power cores at it may be of use.
·Is this something that can be tuned/improved using Tomcat’s NIO settings?
·Is this a bug in Tomcat 7 (websocket/NIO)?
·Are we hitting some issue with 1000+ highly active NIO sockets?
The nature of this application is that it’s primary functionality is to
process binary messages on (ideally 10 – 20K websockets), forward them
to another server for processing, then send the responses back. As a
result, the performance of the websocket communication is a critical
factor for success.
Please let me know if you’d like to discuss this – I’m available any time.
Thanks,
Bob
We have been profiling this in YourKit and as we increase the number of
websockets, naturally we see the CPU utilization increase. The concern
is that we’re seeing this at what seems to be a fairly low number of
websockets. The YourKit profiling showed 41% of the time being spent in
the following code. My apologies for the formatting. I can provide an
attachment out-of-band if it would be helpful.
|
+---org.apache.tomcat.util.net.NioEndpoint$Poller.run()
| 38,594 41 % | 0 |
| | |
|
| |
| |
+---sun.nio.ch.SelectorImpl.select(long)
| 25,615 27 % | 0 |
| | | |
| | |
| | |
+---sun.nio.ch.SelectorImpl.lockAndDoSelect(long)
| 25,615 27 % | 0 |
| | |
|
| | |
| | |
+---sun.nio.ch.WindowsSelectorImpl.doSelect(long)
| 25,615 27 % | 46 |
| | |
|
| | |
| | |
+---sun.nio.ch.WindowsSelectorImpl.resetWakeupSocket()
| 13,681 14 % | 1,107 |
| | |
|
| | |
| | |
+---sun.nio.ch.WindowsSelectorImpl$StartLock.access$300(WindowsSelectorImpl$StartLock)
| 10,826 11 % | 0 |
| | |
|
| | |
| | |
+---sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys()
| 1,029 1 % | 0 |
| | | |
|
| |
| | |
+---sun.nio.ch.SelectorImpl.processDeregisterQueue()
| 31 0 % | 0 |
| |
|
| | |
| |
+---org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(SelectionKey,
NioEndpoint$KeyAttachment) | 11,996 13 % | 31 |
| | |
|
| | |
| | |
+---org.apache.tomcat.util.net.NioEndpoint.processSocket(NioChannel,
SocketStatus, boolean) | 11,778 12 % | 0 |
| | | |
|
| | |
| | | |
+---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable)
| 11,746 12 % | 0 |
| | | | | |
| |
|
| | | | |
+---org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(Runnable,
long, TimeUnit) | 11,746 12 % | 0 |
| | | | |
|
| | |
| | | | |
+---java.util.concurrent.ThreadPoolExecutor.execute(Runnable)
| 11,746 12 % | 15 |
| | | | |
|
| | |
| | | | |
+---org.apache.tomcat.util.threads.TaskQueue.offer(Object)
| 11,731 12 % | 0 |
| | | | |
|
| | |
| | | | |
+---org.apache.tomcat.util.threads.TaskQueue.offer(Runnable)
| 11,731 12 % | 46 |
| | | | |
|
| | |
| | | | |
+---java.util.concurrent.LinkedBlockingQueue.offer(Object)
| 11,668 12 % | 15 |
*Bob DeRemer*
*Senior Director, Architecture and Development*
Description: Description: Description: Description:
cid:image001.png@01CBE3DE.51A12030
http://www.thingworx.com <http://www.thingworx.com/>
Skype: bob.deremer.thingworx
O: 610.594.6200 x812
M: 717.881.3986
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
