Re: stopping robots

2006-07-31 Thread Marc Espie
I've got a robots.txt, and a script that loops to infinity. Actually, it's a useful page on the server, there's a list that can be ordered two ways, and switching from one to the other increments a parameter at the end of the invocation. A robot has no business reading that specific page in the fi

Re: stopping robots

2006-07-26 Thread Nick Guenther
On 7/25/06, Mike Erdely <[EMAIL PROTECTED]> wrote: prad wrote: > what is the best way to stop those robots and spiders from getting in? Someone on this list (who can reveal themselves if they want) has a pretty good setup to block "disrespectful" robots. They have a robots.txt file that specifi

Re: stopping robots

2006-07-25 Thread Spruell, Darren-Perot
From: [EMAIL PROTECTED] > what is the best way to stop those robots and spiders from getting in? > > .htaccess? > robot.txt and apache directives? > find them on the access_log and block with pf? > > i should also ask whether it is a good idea to block robots > in the first place > since some

Re: stopping robots

2006-07-25 Thread Mike Erdely
prad wrote: what is the best way to stop those robots and spiders from getting in? Someone on this list (who can reveal themselves if they want) has a pretty good setup to block "disrespectful" robots. They have a robots.txt file that specifies a "Disallow: /somedir/". Anyone that actually

Re: stopping robots

2006-07-25 Thread Han Boetes
I got these tips from an old message on this list, I hope this helps you as well. # rule-based rewriting engine to rewrite requested URLs on the fly LoadModule rewrite_module /usr/lib/apache/modules/mod_rewrite.so # # Redirect allows you to tell clients about documents which used to exist in

Re: stopping robots

2006-07-25 Thread Rogier Krieger
On 7/25/06, prad <[EMAIL PROTECTED]> wrote: what is the best way to stop those robots and spiders from getting in? The sure way to stop robots and spiders is to shut down your web server. I don't suppose that's the answer you're looking for. Treat malicious robots as malicious/unwelcome users.

Re: stopping robots

2006-07-25 Thread Darrin Chandler
On Tue, Jul 25, 2006 at 02:45:28PM -0700, prad wrote: > what is the best way to stop those robots and spiders from getting in? > > .htaccess? > robot.txt and apache directives? > find them on the access_log and block with pf? > > i should also ask whether it is a good idea to block robots in the

Re: stopping robots

2006-07-25 Thread Jack J. Woehr
On Jul 25, 2006, at 3:45 PM, prad wrote: > which are good robots and which are bad? The good ones are Asenion robots, the bad ones are non-Asenion robots. But that's not a hard-and-fast rule; remember the Nestor series. --- Jack J. Woehr Director of Development Absolute Performance, Inc. [EMAI

stopping robots

2006-07-25 Thread prad
what is the best way to stop those robots and spiders from getting in? .htaccess? robot.txt and apache directives? find them on the access_log and block with pf? i should also ask whether it is a good idea to block robots in the first place since some do help to increase presence on the web. whi