Re: Bloking Bad bots

[lists]

I'd be shocked if the map function doesn't use a smart search scheme rather than check every item.   From: Anoop Alias Sent: Monday, November 14, 2016 7:41 AM To: Nginx Reply To: nginx@nginx.org Subject: Re: Bloking Bad bots I had asked the same question once and got no to the point response. So here is what I infer: the if causes nginx to check the header for each request against the list of patterns you have configured and return a 403 if found .  So the processing slows down on each request to for the if processing..  If you see mod_security etc ..this is also doing something similar and doing a check on each request - so in that way (that is if you are willing to compromise lack of speed for the user agent checking) this is fine . But you are definitely making the nginx slower and consume more resource by adding the if  there and making it more by increasing the list size. On Mon, Nov 14, 2016 at 9:00 PM, <li...@lazygranch.com> wrote: You can block some of those bots at the firewall permanently.   I use the nginx map feature in a similar manner, but I don't know if map is more efficient than your code. ‎I started out blocking similar to your scheme, but the map feature looks clear to me in the conf file. Majestic and Sogou sure are annoying. For what I block, I use 444 rather than 403. (And yes, I know that destroys the matter/anti-matter mix of the universe, so don't lecture me.) I then eyeball the 444 hits periodically, using a script to pull the 444 requests out of the access.log file. I have another script to get just the IP addresses from access.log. For the search engines like Majestic and Sogou, which don't seem to have an IP space you can look up via BGP tools, I take the IP used and add it to my firewall blocking table. I can go weeks before a new IP gets used.   Original Message   From: debilish99 Sent: Monday, November 14, 2016 7:04 AM To: nginx@nginx.org Reply To: nginx@nginx.org Subject: Bloking Bad bots Hello, I have a server with several domains, in the configuration file of each domain I have a line like this to block bad bots. If ($ http_user_agent ~ * (zealbot|MJ12bot|AhrefsBot|sogou|PaperLiBot|uipbot|DotBot|GetIntent|Cliqzbot|YandexBot|Nutch|TurnitinBot|IndeedBot) Return 403; } This works fine. The question is, if I increase the list of bad bots to 1000, for example, this would be a speed problem when nginx manages every request that arrives. I have domains that can have 500,000 hits daily and up to 20,000 hits. Thank you all. Greetings. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270930,270930#msg-270930 _______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx _______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx -- Anoop P Alias

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx