Terry J. Reedy <tjre...@udel.edu> added the comment: Because of the line break, clicking that link gives "Server error 404". http://www.robotstxt.org/norobots-rfc.txt works (so please pay attention to formatting). The main page is http://www.robotstxt.org/robotstxt.html
The way I read the grammar, 'records' (which start with an agent line) cannot have blank lines and must be separated by blank lines. Other than than, the suggestion seems reasonable, but it also seems like a feature request. Does test/test_robotparser pass with the patch? I also do not see "Crawl-delay" and "Sitemap" (from whitehouse.gov) in the grammar referenced above. So I wonder if de facto practice has evolved. Philip S.: do you have any opinions? (I am asking you because of your comments on #1437699.) ---------- nosy: +osvenskan, terry.reedy _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue13281> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
