Hi all, Just wonder if Nutch takes into consideration rules from the robots.txt file while crawling a site.
-- Best Regards Alexander Aristov
- robots.txt Alexander Aristov
- Re: robots.txt Erik Hatcher
Hi all, Just wonder if Nutch takes into consideration rules from the robots.txt file while crawling a site.
-- Best Regards Alexander Aristov