[issue36207] robotsparser deny all with some rules

wats0ns added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? -- nosy: +que

[issue36207] robotsparser deny all with some rules

New submission from wats0ns : RobotsParser parse a "Disallow: ?" rule as a deny all, but this is a valid rule that should be interpreted as "Disallow: /?*" or "Disallow: /*?*" -- components: Library (Lib) messages: 337285 nosy: quentin-maire priority:

[issue36207] robotsparser deny all with some rules

wats0ns added the comment: I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in mo