Andre Burgaud <andre.burg...@gmail.com> added the comment:
During testing identified a related issue that is fixed by the same sort function implemented to address the longest match rule. This related problem also addressed by this change takes into account the situation when 2 equivalent rules (same path for allow and disallow) are found in the robotstxt. In such a situation allow should be used: https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.2 ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue39187> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
