On Saturday 09 November 2019 10:10:53 Andy Smith wrote: > Hello, > > On Sat, Nov 09, 2019 at 08:43:25AM -0500, Gene Heskett wrote: > > I've done that with the help of a previous responder and now have > > 99% of the pigs that ignore my robots.txt blocked. semrush is > > extremely determined and has switched to a 4th address I've not seen > > before, but is no longer DDOSing my site. > > You've repeatedly been advised to block these bots in Apache by > their UserAgent. Have you tried that yet? It would be a lot simpler > than fail2ban or trying to keep up with their IP addresses. > Maybe, but semrush has a variation in the user agent spelling that makes a block of xx.xx.xx.xx/24 more effective. I am now adding rules to block the whole /24 for some of the more obnoxious. bytespider in fact needs a /16, they've apparently 2 whole /24 blocks full of bots.
> Regards, > Andy Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page <http://geneslinuxbox.net:6309/gene>
