abhinav-phi commented on PR #2121: URL: https://github.com/apache/stormcrawler/pull/2121#issuecomment-5574580174
Thanks — pushed 6f2730ba: - **Values below -1:** a `http.content.limit` in the metadata below -1 is now ignored as a configuration error (`metadataLimit >= -1` guard). - **Comment contradiction:** kept the widening, reworded the comment to say it outright — the robots.txt fetch is deliberately allowed a larger read than pages when `http.content.limit` is below the 512 kiB RFC floor, because that floor is the reason `http.robots.content.limit` exists. - **Other protocols:** confirmed — only the okhttp `HttpProtocol` reads the `http.content.limit` *metadata* key (`HttpProtocol.java:442`). `WARCSpout` reads the *config* value directly (`WARCSpout.java:373`) and is unaffected, as you noted. The robot rules parser only writes the key into `fetchRobotsMd`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
