Hello Mathias,
Mathias Walter wrote:
I don't know it exactly. The problem is that the sites are linked from
anywhere. I'm not sure, if a crawler that follows the link
http://mydomain:port/servlet/page.jsp, looks for the robots.txt in the ROOT
webapp.
Just last week we've installed a robots.txt
Hi,
> > I've put a robots.txt in webapps/ROOT, but this file is not
> > read again.
>
> So, to check, the crawlers are not reading your robots.txt
> and are crawling your site anyway?
I don't know it exactly. The problem is that the sites are linked from
anywhere. I'm not sure, if a crawler tha
> From: Mathias Walter [mailto:[EMAIL PROTECTED]
> How can I prevent crawler and bots according to their user agent?
>
> I've put a robots.txt in webapps/ROOT, but this file is not
> read again.
So, to check, the crawlers are not reading your robots.txt and are crawling
your site anyway?
> I'd l
Hi,
I'm using Tomcat without the Apache Web Server. Now, some bots are crawling
my database driven website.
How can I prevent crawler and bots according to their user agent?
I've put a robots.txt in webapps/ROOT, but this file is not read again. I'd
like to stop crawlers by their useragent strin
