On Mon, Nov 21, 2005 at 10:35:01AM -0500, Joey Hess wrote: > Brian M. Carlson wrote:
>> Additionally, the robots.txt is being violated by "bts cache", so >> perhaps someone should file a bug. > "bts cache" is not a web spider, it only downloads the bugs that you > tell it to. It violates the robots.txt much less than eg, wget. wget (in non-recursive behaviour, never following any link) downloads only the URLs you tell it too, so according to the same argument, doesn't violate robots.txt . In recursive mode (following links), wget respects robots.txt (by default). -- Lionel -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
