Thirumani S saw fit to inform LIH that:
>I observed a peculiar behavior while using kwebget. No matter whatever
>file I ask it to probe to more depth and get the links there is always
>this robots.txt popping up. I wonder how it can be there in all the web
>pages. What could be wrong?
robots.txt is a file which can be placed by a webmaster in any
directory. It basically means "no spiders or robots" - that is, search
engine bots, wget downloads etc - allowed.
Get it?
--
Suresh Ramasubramanian | President, CAUCE India
[EMAIL PROTECTED] | [EMAIL PROTECTED]
http://www.india.cauce.org | Stopping Spam In India
--
Calm down, it's *only* ones and zeroes.
-----------------------------------------------------------------------
LIH is all for free speech. But it was created for a purpose - to help
people discuss issues about installing and running Linux. If your
messages are counterproductive to this purpose, your privileges to
submit messages can and will be revoked.
