Hi Mike,
It certainly sounds like a crawler/bot getting stuck in a loop. In your log
there, I see the client IP address 190.92.203.86, which belongs to Huawei Cloud
Singapore. I've seen a lot of bots/crawlers from Huawei Cloud Singapore hitting
Australian Koha sites over the last 6 months or so.
That 'AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha' error
is interesting. If you look at /etc/apache2/conf-enabled/serve-cgi-bin.conf,
you'll see that at a global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So
if the crawler sent any HTTP requests using your IP address and not the
hostname, they'd be caught by that directive instead of your name-based virtual
host. Could be some other explanations for why the virtual host wasn't used,
but overall that would explain that message.
Anyway, it's not necessarily a Koha-specific issue. If you're not already using
it, I'd suggest you look at installing and setting up something like fail2ban.
That said, I have noticed the bots out of Huawei Cloud Singapore tend to cycle
through a lot of different IP addresses, which does make things tricky.
Sometimes, it'll just use one IP address that is easy to detect and block, but
sometimes it might just do 1-2 hits per IP address (from a variety of different
IP ranges).
Let me know if you'd like to chat more about it.
David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
-----Original Message-----
Date: Sat, 13 Jul 2024 21:10:36 +1000
From: Mike Lake <mi...@speleonics.com.au>
To: koha@lists.katipo.co.nz
Subject: Re: [Koha] Out of memory when Koha starts due to
opac-search.pl and 500.pl
Message-ID: <f034d85a454901421773c0f4df4a0...@speleonics.com.au>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi
Katrin suggested:
> it might be that you are hit by a bad crawler/bot
Thanks Katrin. That *may* have been the cause. The system is working OK
at present. I did a complete shutdown and reboot.
I did notice in the opac-error.log, which is now over 10 MB, a recurring
query (see below) that was being made every 30 seconds. Exact same
query, clearly automated. That seems to have ended now.
cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available
I was also getting these errors which were filling up the logs:
[Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client
190.92.203.86:51260]
AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha
There is no such perl script
$ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
so I just created one to return "hello".
Now our Koha instance is back up again and our VM is coping with the
load. https://opac.caves.org.au
Thanks for the reply.
I'll make another separate post on another current opac-error.log error
line, if it still persists, after I upgrade from 23.11.05
Mike
ASF Sys Admin
On 2024-07-13 7:34 pm, Katrin Fischer wrote:
> Hi Mike,
>
> it might be that you are hit by a bad crawler/bot and need to block
> access for them in your firewall. There are some that ignore the
> robots.txt and they can bring down a Koha server.
>
> I you look at the Apache access logs you might see that all those
> requests come from the same IP address.
>
> Hope this helps,
>
> Katrin
>
> On 10.07.24 13:02, Mike Lake wrote:
>> Hi all
>>
>> I'm having serious problems with my Koha instance. It serves the OPAC
>> for the Australian Speleological Federation. We are currently on Koha
>> 23.11 on a Debian 10.13. The system has been running fine for ages.
>>
>> I was getting errors from the OOM killer:
>>
>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
>> file-rss:0kB, shmem-rss:0kB
>> opac-search.pl invoked oom-killer:
>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
>> oom_score_adj=0
>> opac-search.pl cpuset=/ mems_allowed=0
>>
>> So I shutdown Koha (took a while as I was out of memory)
>> systemctl stop koha-common.service
>>
>> Rebooted the machine and when i bought Koha up:
>> systemctl start koha-common.service
>> Now I'm still getting 96 processes & errors taking all CPU and memory:
>>
>> 3620 R /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
>> 3622 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3624 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3625 D /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3627 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3629 D /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3630 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3633 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>
>> Actually its 96 x opac-search.pl + 57 x 500.pl
>>
>> A reboot does not help. Every time I start Koha those processes appear
>> and take all cores and memory.
>>
>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
>> database server."
>> Attempts to start it: systemctl start mariadb.service
>> give that error probably because I'm out of memory does to the 100
>> perl processes running.
>>
>> A "systemctl stop koha-common.service" does not stop or end those
>> opac-search.pl or 500.pl processes.
>>
>> The /var/log/koha/opac/opac-error.log says:
>>
>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
>> output before headers: 500.pl
>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
>> output before headers: opac-search.pl
>>
>> Something is borked :-( Help most welcome.
>>
> _______________________________________________
>
> Koha mailing list http://koha-community.org
> Koha@lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
--
Mike
------------------------------
Subject: Digest Footer
_______________________________________________
Koha mailing list
Koha@lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
End of Koha Digest, Vol 225, Issue 8
************************************
_______________________________________________
Koha mailing list http://koha-community.org
Koha@lists.katipo.co.nz
Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
