Hello,
On Sat, Nov 09, 2019 at 01:34:11PM -0500, Gene Heskett wrote:
> On Saturday 09 November 2019 10:10:53 Andy Smith wrote:
> > You've repeatedly been advised to block these bots in Apache by
> > their UserAgent. Have you tried that yet? It would be a lot simpler
> > than fail2ban or trying to keep up with their IP addresses.
> >
> Maybe, but semrush has a variation in the user agent spelling that makes
> a block of xx.xx.xx.xx/24 more effective.
Really?
$ cat /var/log/apache2/access.log{,.1} | awk -F '[()]' 'tolower($0) ~ /semrush/
{ print $2 }' | sort | uniq -c | sort -rn
95 compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html
80 compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html
29 compatible; SemrushBot-BA; +http://www.semrush.com/bot.html
I'll suggest once more just blocking UserAgents that match
"SemrushBot" but I realise I am just howling into the void.
Regards,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
