Jumping in a bit late. But very recently I also saw quite a bit of traffic from Singapore (Huawei). Using about ten different ip ranges (x.x.0.0/16) and lots of different IPs. So blocking is hard. If you use nginx, rate limiting might be a good option to explore. I added a rate limit too on the x.x of the IP address.
Op ma 15 jul 2024 om 15:22 schreef Mike Lake <mi...@speleonics.com.au>: > Hi Davis and all > > Ah :-) Some very good help there. Yes I did some whois queries and many > are from Singapore. > Also it had not realised that there is an alias "ScriptAlias /cgi-bin/ > /usr/lib/cgi-bin/" as I had never looked at serve-cgi-bin.conf > And yes why would anyone use an IP address to make a Koha query. I > didn't realise that would hit that script alias then. > > I'm using fail2ban but up till now just for SSH. So tonight I have been > looking at a regex for Apache to match some of the errors in the Koha > logs. > > I'll get back with how I go. Regexes :-( > > Thanks :-) > Mike Lake > > > On 2024-07-15 9:49 am, David Cook wrote: > > Hi Mike, > > > > It certainly sounds like a crawler/bot getting stuck in a loop. In your > > log there, I see the client IP address 190.92.203.86, which belongs to > > Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei > > Cloud Singapore hitting Australian Koha sites over the last 6 months or > > so. > > > > That 'AH02811: script not found or unable to stat: > > /usr/lib/cgi-bin/koha' error is interesting. If you look at > > /etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a > > global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the > > crawler sent any HTTP requests using your IP address and not the > > hostname, they'd be caught by that directive instead of your name-based > > virtual host. Could be some other explanations for why the virtual host > > wasn't used, but overall that would explain that message. > > > > Anyway, it's not necessarily a Koha-specific issue. If you're not > > already using it, I'd suggest you look at installing and setting up > > something like fail2ban. That said, I have noticed the bots out of > > Huawei Cloud Singapore tend to cycle through a lot of different IP > > addresses, which does make things tricky. Sometimes, it'll just use one > > IP address that is easy to detect and block, but sometimes it might > > just do 1-2 hits per IP address (from a variety of different IP > > ranges). > > > > Let me know if you'd like to chat more about it. > > > > David Cook > > Senior Software Engineer > > Prosentient Systems > > Suite 7.03 > > 6a Glen St > > Milsons Point NSW 2061 > > Australia > > > > Office: 02 9212 0899 > > Online: 02 8005 0595 > > > > -----Original Message----- > > > > Date: Sat, 13 Jul 2024 21:10:36 +1000 > > From: Mike Lake <mi...@speleonics.com.au> > > To: koha@lists.katipo.co.nz > > Subject: Re: [Koha] Out of memory when Koha starts due to > > opac-search.pl and 500.pl > > Message-ID: <f034d85a454901421773c0f4df4a0...@speleonics.com.au> > > Content-Type: text/plain; charset=UTF-8; format=flowed > > > > Hi > > > > Katrin suggested: > >> it might be that you are hit by a bad crawler/bot > > > > Thanks Katrin. That *may* have been the cause. The system is working OK > > at present. I did a complete shutdown and reboot. > > > > I did notice in the opac-error.log, which is now over 10 MB, a > > recurring > > query (see below) that was being made every 30 seconds. Exact same > > query, clearly automated. That seems to have ended now. > > > > cgi-bin/koha/ > opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available > > > > I was also getting these errors which were filling up the logs: > > > > [Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client > > 190.92.203.86:51260] > > AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha > > > > There is no such perl script > > $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/' > > so I just created one to return "hello". > > > > Now our Koha instance is back up again and our VM is coping with the > > load. https://opac.caves.org.au > > > > Thanks for the reply. > > I'll make another separate post on another current opac-error.log error > > line, if it still persists, after I upgrade from 23.11.05 > > > > Mike > > ASF Sys Admin > > > > On 2024-07-13 7:34 pm, Katrin Fischer wrote: > >> Hi Mike, > >> > >> it might be that you are hit by a bad crawler/bot and need to block > >> access for them in your firewall. There are some that ignore the > >> robots.txt and they can bring down a Koha server. > >> > >> I you look at the Apache access logs you might see that all those > >> requests come from the same IP address. > >> > >> Hope this helps, > >> > >> Katrin > >> > >> On 10.07.24 13:02, Mike Lake wrote: > >>> Hi all > >>> > >>> I'm having serious problems with my Koha instance. It serves the OPAC > >>> for the Australian Speleological Federation. We are currently on > >>> Koha > >>> 23.11 on a Debian 10.13. The system has been running fine for ages. > >>> > >>> I was getting errors from the OOM killer: > >>> > >>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, > >>> file-rss:0kB, shmem-rss:0kB > >>> opac-search.pl invoked oom-killer: > >>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, > >>> oom_score_adj=0 > >>> opac-search.pl cpuset=/ mems_allowed=0 > >>> > >>> So I shutdown Koha (took a while as I was out of memory) > >>> systemctl stop koha-common.service > >>> > >>> Rebooted the machine and when i bought Koha up: > >>> systemctl start koha-common.service > >>> Now I'm still getting 96 processes & errors taking all CPU and > >>> memory: > >>> > >>> 3620 R /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl > >>> 3622 R /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl > >>> 3624 R /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl > >>> 3625 D /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl > >>> 3627 R /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl > >>> 3629 D /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl > >>> 3630 R /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl > >>> 3633 D /usr/bin/perl > >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl > >>> > >>> Actually its 96 x opac-search.pl + 57 x 500.pl > >>> > >>> A reboot does not help. Every time I start Koha those processes > >>> appear > >>> and take all cores and memory. > >>> > >>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 > >>> database server." > >>> Attempts to start it: systemctl start mariadb.service > >>> give that error probably because I'm out of memory does to the 100 > >>> perl processes running. > >>> > >>> A "systemctl stop koha-common.service" does not stop or end those > >>> opac-search.pl or 500.pl processes. > >>> > >>> The /var/log/koha/opac/opac-error.log says: > >>> > >>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script > >>> output before headers: 500.pl > >>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script > >>> output before headers: opac-search.pl > >>> > >>> Something is borked :-( Help most welcome. > >>> > >> _______________________________________________ > >> > >> Koha mailing list http://koha-community.org > >> Koha@lists.katipo.co.nz > >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha > > > > -- > > Mike > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > Koha mailing list > > Koha@lists.katipo.co.nz > > https://lists.katipo.co.nz/mailman/listinfo/koha > > > > > > ------------------------------ > > > > End of Koha Digest, Vol 225, Issue 8 > > ************************************ > > -- > Mike > _______________________________________________ > > Koha mailing list http://koha-community.org > Koha@lists.katipo.co.nz > Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha > _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
