Mher Movsisyan added the comment: The second patch only normalizes the url. From http://www.robotstxt.org/norobots-rfc.txt it is not clear how to handle multiple rules with the same prefix.
---------- Added file: http://bugs.python.org/file29476/parser2.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue17403> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
