> Can you confirm whether or not the issue exists with 6.0.26 and 1.1.22?
I cannot. We have tried repeatedly to reproduce this problem in a test environment where such experimentation is tolerated, but the problem simply does not manifest using available load testing tools. We attempted to try 7.0.23+1.2.20 in production, but I couldn't get a working configuration for the supported SSL/TLS protocols we require. > How long do the periods of high CPU usage last? Varies quite a bit. Based on data from our enterprise monitoring system, the problem lasts as little as 5 min or less and has lasted as much as 25 min. YourKit data from the incidents that have happened in the past 24h shows similarly wide variation: 2m40s 1m38s 13m11s 8s > How sure are you that the CPU is being burned in the threads that are in > sendbb? I'm certain of nothing, but I can share the evidence. YourKit CPU profiling data shows that request processing threads consume the majority of resources during these periods. No surprise there. However, the only method that is consistently executed during these periods is sendbb() by two or more threads. For the last problem period, which happens to be the shortest (8s), catalina-exec threads accounted for 89% of CPU use. Following are the RUNNABLE threads at 1s sample intervals during that period: Stacks at 09:11:01 AM (uptime 1 day 3:26:47) 1. catalina-exec-45 [RUNNABLE, IN_NATIVE] CPU time: 1:57 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:32 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:02 AM (uptime 1 day 3:26:48) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:21 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:33 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:03 AM (uptime 1 day 3:26:49) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:22 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:34 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:04 AM (uptime 1 day 3:26:50) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:23 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:05 AM (uptime 1 day 3:26:51) 1. catalina-exec-38 [RUNNABLE, IN_NATIVE] CPU time: 1:28 java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) 2. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:24 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 3. catalina-exec-44 [RUNNABLE] CPU time: 1:43 java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) 4. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:06 AM (uptime 1 day 3:26:52) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:36 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:07 AM (uptime 1 day 3:26:53) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:37 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 3. catalina-exec-52 [RUNNABLE] CPU time: 0:04 java.lang.Throwable.fillInStackTrace() Stacks at 09:11:08 AM (uptime 1 day 3:26:54) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:26 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:38 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:09 AM (uptime 1 day 3:26:55) 1. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:39 org.apache.tomcat.jni.Socket.sendbb(long, int, int) After the last sample, CPU usage drops precipitously. Our application is a Web SSO product that writes relatively small response payloads. It's hard to explain two request processing threads that take several seconds to write small response payloads without citing a bug in the servlet container. M --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
