From: [EMAIL PROTECTED] > what is the best way to stop those robots and spiders from getting in? > > .htaccess? > robot.txt and apache directives? > find them on the access_log and block with pf? > > i should also ask whether it is a good idea to block robots > in the first place > since some do help to increase presence on the web. > which are good robots and which are bad?
And here I've never considered them a threat. Do you have information in a robots.txt that you shouldn't? What's your concern with them? DS
