On 2019-11-09, john doe <johndoe65...@mail.com> wrote: > > Note that using IPs directly is an red herring; you need to use other > means (UserAgent ...) to identify those bots.
Over at semrush they advise the following (with robots.txt in the top directory of the server): To stop SEMrushBot from crawling your site, add the following rules to your "robots.txt" file: To block SEMrushBot from crawling your site for web graph of links, add: User-agent: SemrushBot Disallow: / Please note that there might be a delay up to two weeks before SEMrushBot discovers the changes you made to robots.txt. To remove SEMrushBot from crawling your site for different SEO and technical issues, add: User-agent: SemrushBot-SA Disallow: / They note that it may require up to two weeks for the cure to take effect. > By the sound of it, you cleerly need to learn the httpd server you are > using, then if it is not enough, add fail2ban and iptables into the mix. > > -- > John Doe > > -- “The cradle rocks above an abyss, and common sense tells us that our existence is but a brief crack of light between two eternities of darkness.” "Speak, Memory," Vladimir Nabokov
