On 10.08.2016 17:46, Christopher Schultz wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mike,
On 8/10/16 6:15 AM, Mike Noordermeer wrote:
Hi,
After an upgrade to Tomcat 8.5, we are experiencing an issue where
Tomcat starts generating a high CPU load (100%), probably after an
HTTP network scan. The bug seems to be related to Windows, NIO and
possibly SSL. I have a Yourkit dump and several thread dumps that
show the issue, and was wondering if anyone is interested in this,
and if we can gather any extra information to help debug this
issue.
Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO
JSSE HTTPS connector.
Out of nothing, Tomcat starts using 100% CPU. I made some thread
dumps, available here:
https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc
https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02
https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52
https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a
https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
I also have some Yourkit stats available, but as these may contain
confidential information, I won't share them in public. Basically,
what we see is that the thread https-jsse-nio-443-ClientPoller-0
is continuously runnable and using CPU on
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various
other https-jsse-nio-443-exec threads are waiting (parked) or
running. These threads together take up all the CPU. A Yourkit
thread view showing the issue starting around 11:02:
https://dl.eveoh.nl/yc_fal.png
We _suspect_ the issue is triggered by an HTTP scan, which
generates the following requests in the access log, but we are
still trying to confirm this:
https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
Also, we are trying to confirm whether or not NIO2 shows the same
behaviour.
The behaviour seems to be the same as in this tomcat-users thread:
https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C
CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w...@mail.gmail.com%3E
A similar issue is mentioned for some other products, but I'm not
sure if there's a relation:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240
https://developer.jboss.org/thread/240618?start=0&tstart=0
https://github.com/netty/netty/issues/3857
Our next steps are:
- Switching the production site to NIO2, to see if that fixes the
issue - Checking if we can reproduce the issue by triggering the
HTTP vulnerability scan manually
Any ideas or requests for more information are more than welcome.
Are you fronting with a web server/reverse proxy? Those "-" requests
looks suspiciously like the kinds of requests that Apache httpd makes
to itself to verify that worker threads are still available for
certain things.
Maybe that's a way that HTTP scanners are trying to avoid detection:
by looking like "normal" stuff in the logs.
I'm curious... why are the requests coming from "10.xxx"... isn't that
within your own network? Shouldn't you KNOW what that stuff is?
In-house webserver monitoring software ? (Nagios e.g.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
