mvolikas commented on issue #1597: URL: https://github.com/apache/stormcrawler/issues/1597#issuecomment-3941402217
> We’re now seeing some failures in BasicURLNormalizerTest because URI enforces RFC 3986 compliance strictly, whereas URL is more tolerant of invalid input. We need to decide how we want to handle this going forward. In practice, a crawler may encounter malformed URIs but still attempt to normalize them into a valid URL. The open question is how much recovery or sanitization we want to perform versus simply failing when an invalid URI is encountered. This is a question of how many malformed URI cases we will handle/normalize in BasicURLNormalizer correct? Because edge case replacements can also be externally configured in the `default-regex-normalizers.xml`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
