If robots.txt was getting updated properly, all of our Google search bar problems would be solved. We could then stop telling Google to restrict the search results to a patrticular version from the search box itself. The robots.txt file only allows the current stable docs to be indexed.
If robots.txt was getting updated properly, then only 2.16 should appear in search results, and would continue to appear for a little while after we release 2.18. Then, the next time the Google bot visits lilypond.org, it would read robots.txt, remove the 2.16 docs from the search results, then follow every link from the homepage that isn't "disallowed", i.e. the entire website, with only the 2.18 docs. That's exactly what we want, so let's just fix the problem with robots.txt. By the way, fixing that would kill 3 items in the tracker with one blow: Issue 2909: Manual search returns results from wrong version http://code.google.com/p/lilypond/issues/detail?id=2909 Issue 3209: Searching stable release documentation should only return results from stable release http://code.google.com/p/lilypond/issues/detail?id=3209 Issue 3367: Web/Docs: LilyPond version is not clear on docs web pages http://code.google.com/p/lilypond/issues/detail?id=3367 - Mark ----- Original Message ----- > From: Mark Polesky <markpole...@yahoo.com> > To: bug-lilypond <bug-lilypond@gnu.org> > Cc: > Sent: Friday, June 28, 2013 11:37 AM > Subject: robots.txt in git and online are not the same > > For some reason, http://www.lilypond.org/robots.txt differs > from Documentation/web/server/robots.txt as found in git. > It looks as if it isn't being updated on the server? I > don't get it, but it's worth looking into, since this helps > remove outdated links when someone googles us from the > outside. > > - Mark > > > _______________________________________________ > bug-lilypond mailing list > bug-lilypond@gnu.org > https://lists.gnu.org/mailman/listinfo/bug-lilypond > _______________________________________________ bug-lilypond mailing list bug-lilypond@gnu.org https://lists.gnu.org/mailman/listinfo/bug-lilypond
