rzo1 commented on code in PR #2077:
URL: https://github.com/apache/stormcrawler/pull/2077#discussion_r3879541808
##########
core/src/main/java/org/apache/stormcrawler/protocol/HttpRobotRulesParser.java:
##########
@@ -44,6 +44,8 @@ public class HttpRobotRulesParser extends RobotRulesParser {
protected boolean allow5xx = false;
+ protected boolean allowCrossHostRedirects = false;
Review Comment:
Exempted, one way only. `isSchemeUpgrade` follows http -> https on the same
host when the port is unchanged or both sides use their scheme default, so
`http://h/robots.txt -> https://h/robots.txt` and the `:443` /
explicit-same-port variants are followed by default; `http://h:8080 ->
https://h:9443` stays refused. The downgrade is deliberately not exempt, since
it would put the configured `http.basicauth.*` / `http.custom.headers` on the
wire in clear and let rules read over plain text replace the ones the redirect
was reached from. Also renamed the flag to
`http.robots.redirect.crossorigin.allow` while it is still unreleased — it
governs scheme and port as well as host, and one scheme change is no longer
cross-anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
