Thanks all. OK, I can live with that. So things like Tabulator, Sig.ma and SemWeb Browsers can be expected to go through a general robots.txt Disallow, which is what I was hoping.
Yes, thanks Aidan, I know I can do various User-agents, but I really just wanted to stop anything like googlebot. By the way, have I got my robots.txt right? http://ibm.rkbexplorer.com/robots.txt In particular, is the User-agent: LDSpider correct? Should I worry about case-sensitivity? Thanks again, all. Hugh On 27 Jul 2014, at 19:23, Gannon Dick <[email protected]> wrote: > > -------------------------------------------- > On Sat, 7/26/14, [email protected] <[email protected]> wrote: > > The difference in opinion remains to what extent Linked Data > agents need to pay attention to the robots.txt file. > > As many others have suggested, I buy into the idea of any > agent not relying document-wise on user input being subject to > robots.txt. > > ===================== > +1 > Just a comment. > > Somewhere, sometime, somebody with Yahoo Mail decided that public-lod mail > was spam, so every morning I dig it out because I value the content. > > Of course, I could wish for a Linked Data Agent which does that for me, but > that would be to complete a banal or vicious cycle, depending on the circle > classification scheme in use. I'm looking gor virtuous cycles and in the > case of robots.txt, "The lady doth protest too much, methinks". > --Gannon > > > -- Hugh Glaser 20 Portchester Rise Eastleigh SO50 4QS Mobile: +44 75 9533 4155, Home: +44 23 8061 5652
