[issue36207] robotsparser deny all with some rules
adiboo adib added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: https://www.st-info.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? -- nosy: +adiboo67 ___ Python tracker <https://bugs.python.org/issue36207> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36207] robotsparser deny all with some rules
adiboo adib added the comment: Hi now it work on all my website https://www.matelesecretairemedicale.com/ -- ___ Python tracker <https://bugs.python.org/issue36207> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
