Re: robots.txt

[Erik Hatcher]

On Oct 20, 2008, at 8:58 AM, Alexander Aristov wrote:

Just wonder if Nutch takes into consideration rules from the robots.txt file

while crawling a site.

Wrong e-mail list, but yeah, Nutch supports robots.txt considerations.

        Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]