On 17/02/2016 at 01:39, Jason Morris wrote:
> What would be the proper config of robots.txt to allow only Google to crawl a
> JSPWiki installation and nothing else?
It's all in Google's robots.txt documentation
https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
However
Can you clarify, do you mean:-
Only allow GoogleBot?
Only crawl JSPWiki?
Both?
On 17 February 2016 at 01:39, Jason Morris
wrote:
> Hi All,
>
> What would be the proper config of robots.txt to allow only Google to
> crawl a JSPWiki installation and nothing else?
>
> Cheers,
> Jason
>
> Jas
Thanks Roland!
-Original Message-
From: Roland Whitehead [mailto:rol...@quru.com]
Sent: Wednesday, 17 February 2016 6:20 PM
To: user@jspwiki.apache.org
Subject: Re: robots.txt files and JSPWiki
On 17/02/2016 at 01:39, Jason Morris wrote:
> What would be the proper config of robots.txt to
Both, yes. (Why bother with Bing and the others?)
Roland's reply was what I was looking for I think.
-Original Message-
From: Paul Uszak [mailto:paul.us...@gmail.com]
Sent: Wednesday, 17 February 2016 11:31 PM
To: user@jspwiki.apache.org
Subject: Re: robots.txt files and JSPWiki
Can you
