hi.. why not block the spiders in the web server environtment . i think it is better than has to put the code in the apps.
is it possible ? because i had read that some web server can block unnecesary request before reach the real web server..like nginx, lighttpd, etc.. On Tue, Jul 12, 2011 at 5:38 PM, Jonas Geiregat <jo...@geiregat.org> wrote: > > > > Hi all, > > > > has someone an effective way to block away bots and spiders? > > There are so many ignoring robots.txt - besides facebook and tons of > > Java/xyz clients there are many more illegal spiders around. > > > > So does anyone have a performance effective way to block them out? > > Or do you think - if performance matters, leave them crawling... > > > > regards > > > > Henrik > > The only thing I can think of is checking the request.META dictionary. It > contains the HTTP_HOST and HTTP_USER_AGENT settings by the client. You could > check if those are valid but of course a spider could fake those values. But > I'm guessing not all spiders fake them. > You could at least filter out those that don't set these values. > > It would probably require lot's of heavy regular expression code (since > there are so many valid client headers) which would be best implemented as a > decorator on each view method. Or you could probably put the code in a > middleware. > Eventually taking out those spiders would slow down your request. > > Again I have no experience on this field, it's just an idea that might be > possible. > > Regards, > > Jonas. > > -- > You received this message because you are subscribed to the Google Groups > "Django users" group. > To post to this group, send email to django-users@googlegroups.com. > To unsubscribe from this group, send email to > django-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-users?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
