Hi Davis and all
Ah :-) Some very good help there. Yes I did some whois queries and many
are from Singapore.
Also it had not realised that there is an alias "ScriptAlias /cgi-bin/
/usr/lib/cgi-bin/" as I had never looked at serve-cgi-bin.conf
And yes why would anyone use an IP address to make a Koha query. I
didn't realise that would hit that script alias then.
I'm using fail2ban but up till now just for SSH. So tonight I have been
looking at a regex for Apache to match some of the errors in the Koha
logs.
I'll get back with how I go. Regexes :-(
Thanks :-)
Mike Lake
On 2024-07-15 9:49 am, David Cook wrote:
Hi Mike,
It certainly sounds like a crawler/bot getting stuck in a loop. In your
log there, I see the client IP address 190.92.203.86, which belongs to
Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei
Cloud Singapore hitting Australian Koha sites over the last 6 months or
so.
That 'AH02811: script not found or unable to stat:
/usr/lib/cgi-bin/koha' error is interesting. If you look at
/etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a
global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the
crawler sent any HTTP requests using your IP address and not the
hostname, they'd be caught by that directive instead of your name-based
virtual host. Could be some other explanations for why the virtual host
wasn't used, but overall that would explain that message.
Anyway, it's not necessarily a Koha-specific issue. If you're not
already using it, I'd suggest you look at installing and setting up
something like fail2ban. That said, I have noticed the bots out of
Huawei Cloud Singapore tend to cycle through a lot of different IP
addresses, which does make things tricky. Sometimes, it'll just use one
IP address that is easy to detect and block, but sometimes it might
just do 1-2 hits per IP address (from a variety of different IP
ranges).
Let me know if you'd like to chat more about it.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
-----Original Message-----
Date: Sat, 13 Jul 2024 21:10:36 +1000
From: Mike Lake <mi...@speleonics.com.au>
To: koha@lists.katipo.co.nz
Subject: Re: [Koha] Out of memory when Koha starts due to
opac-search.pl and 500.pl
Message-ID: <f034d85a454901421773c0f4df4a0...@speleonics.com.au>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi
Katrin suggested:
it might be that you are hit by a bad crawler/bot
Thanks Katrin. That *may* have been the cause. The system is working OK
at present. I did a complete shutdown and reboot.
I did notice in the opac-error.log, which is now over 10 MB, a
recurring
query (see below) that was being made every 30 seconds. Exact same
query, clearly automated. That seems to have ended now.
cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available
I was also getting these errors which were filling up the logs:
[Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client
190.92.203.86:51260]
AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha
There is no such perl script
$ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
so I just created one to return "hello".
Now our Koha instance is back up again and our VM is coping with the
load. https://opac.caves.org.au
Thanks for the reply.
I'll make another separate post on another current opac-error.log error
line, if it still persists, after I upgrade from 23.11.05
Mike
ASF Sys Admin
On 2024-07-13 7:34 pm, Katrin Fischer wrote:
Hi Mike,
it might be that you are hit by a bad crawler/bot and need to block
access for them in your firewall. There are some that ignore the
robots.txt and they can bring down a Koha server.
I you look at the Apache access logs you might see that all those
requests come from the same IP address.
Hope this helps,
Katrin
On 10.07.24 13:02, Mike Lake wrote:
Hi all
I'm having serious problems with my Koha instance. It serves the OPAC
for the Australian Speleological Federation. We are currently on
Koha
23.11 on a Debian 10.13. The system has been running fine for ages.
I was getting errors from the OOM killer:
oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
file-rss:0kB, shmem-rss:0kB
opac-search.pl invoked oom-killer:
gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
oom_score_adj=0
opac-search.pl cpuset=/ mems_allowed=0
So I shutdown Koha (took a while as I was out of memory)
systemctl stop koha-common.service
Rebooted the machine and when i bought Koha up:
systemctl start koha-common.service
Now I'm still getting 96 processes & errors taking all CPU and
memory:
3620 R /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
3622 R /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3624 R /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3625 D /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-search.pl
3627 R /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3629 D /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/opac-search.pl
3630 R /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/errors/500.pl
3633 D /usr/bin/perl
/usr/share/koha/opac/cgi-bin/opac/errors/500.pl
Actually its 96 x opac-search.pl + 57 x 500.pl
A reboot does not help. Every time I start Koha those processes
appear
and take all cores and memory.
I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
database server."
Attempts to start it: systemctl start mariadb.service
give that error probably because I'm out of memory does to the 100
perl processes running.
A "systemctl stop koha-common.service" does not stop or end those
opac-search.pl or 500.pl processes.
The /var/log/koha/opac/opac-error.log says:
[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
output before headers: 500.pl
[cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
output before headers: opac-search.pl
Something is borked :-( Help most welcome.
_______________________________________________
Koha mailing list http://koha-community.org
Koha@lists.katipo.co.nz
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
--
Mike
------------------------------
Subject: Digest Footer
_______________________________________________
Koha mailing list
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
End of Koha Digest, Vol 225, Issue 8
************************************
--
Mike
_______________________________________________
Koha mailing list http://koha-community.org
Koha@lists.katipo.co.nz
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
