James,
On 6/24/24 17:57, James H. H. Lampert wrote:
On 6/24/24 12:03 PM, Tim Funk wrote:
Conversely, this is a good time for the developers to review
their server logging and tune it to be less verbose for these
normal exceptions. As well as implementing logging frameworks
and logging at the appropriate level (fatal through debug)
Thanks for your thoughts on the subject.
The primary problem isn't that the catalina.out file is enormous, nor
that the log messages are too verbose (if anything, they're not verbose
enough: a timestamp would help) ; it's that while the random filenames
were being tried, in quick succession, the CPU usage of the JVM job
skyrocketed, and legitimate traffic was probably having trouble getting in.
Hmm. Are you getting requests for the same (missing) resource over and
over again, or many different ones? I suspect different, since attackers
have a huge dictionary of "telltale resources" to identify a service as
running one of many known-vulnerable services (e.g. Wordpress, JIRA, etc.).
Why aren't you seeing the source-IP in your own logs? If you are using a
reverse-proxy and always getting the IP of the proxy, you should be able
to (a) configure the reverse-proxy to provide its own IP to you and
configure Tomcat to use that for logging and/or (b) look-up the same
request in the proxy logs. Once you have that, you can ban.
Banning can be done in many ways.
One way would be just by-hand. If you see a huge number of requests from
one IP, just ban it. If it's more than one IP, then you get to play
whack-a-mole and automated solutions would be better.
Be careful with fail2ban (which I like a lot, but I've never used it
with HTTP logs): you can easily ban people quickly who aren't actually
attacking you. If you use fail2ban (or similar), I would be very
conservative about its configuration. Basically, you choose the number
of flags that have to be thrown per-IP per-time-interval. So something
like "100 404s in a minute per-IP" would be conservative.
fail2ban is definitely available on Linux and probably also BSD. I think
you are on AS/400 or something like that, right? And of course you'd
want to ban as far up the chain as possible, at the reverse-proxy, TLS
terminator, or load-balancer. I'm not sure exactly what options you have
in that environment. fail2ban works directly with the OS's firewall, so
on Linux that's iptables and on BSD that's generally ipfilter. If AS/400
has a built-in firewall then it's possible fail2ban has a build for it.
Tim's suggestion of a WAF is a good one. I've tried to configure
mod_security for Apache httpd and it's got like 100M rules and it
immediately banned every one of our developers when we used it
out-of-the-box. It is SUPER PARANOID about everything. If you are
considering using mod_security, I would (a) run it only in a development
/ testing environment and (b) run it in "audit mode" for a week or so
and look at all the complaints it makes. I have had to specifically
disable a LARGE number of checks to make it even partially usable.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
