The GitHub Actions job "Java CI with Maven" on stormcrawler.git/main has succeeded. Run started by GitHub user rzo1 (triggered by rzo1).
Head commit for run: 5dd6d99cba62e849cd857edccb260d052b749f29 / Abhinav <[email protected]> File scheme off by default; reads confined to file.protocol.root (#2124) * File scheme off by default; reads confined to file.protocol.root (#2081) crawler-default.yaml enabled the file scheme in every topology built on the library defaults, and FileProtocol read whatever path the worker user could read: a fetched page can put a file:// URL into the frontier and the response body - including the topology configuration - went on to be parsed and indexed. The shipped protocols list is now http,https, and the file scheme has to be enabled deliberately. FileProtocol gains a file.protocol.root key: when it is set, only paths canonicalising below that directory are served (symlinks included), everything else returns 403; when it is not set, the file scheme serves nothing. An operator crawling a local corpus sets both keys. * Refuse file URLs with a host, note the check-vs-read window (#2081) Review feedback on #2124: - a file URL carrying a host component (file://evil.example.com/etc/passwd) is refused outright instead of the host being silently ignored: the host is meaningless for a local read, and accepting it would only invite spellings that look remote - a comment documents the accepted check-vs-read window: canonicalise and check are two operations, so a symlink swapped in between would be followed; that is fine now that the file scheme is opt-in and root-confined, the operator who enables it accepts the worker user's read rights as the boundary - confirmed nothing in the archetypes or the ProtocolFactory assumes file is in the protocols list * Normalize imports and formatting to satisfy CI The import groups are collapsed into the single ASCII-sorted group the project checkstyle config demands (STATIC###THIRD_PARTY_PACKAGE), the constructors in URLFilters are grouped as checkstyle requires, and the touched files are reformatted with the google-java-format version the git-code-format plugin pins (1.35.0, AOSP). No code changes. * Pin the host check and document file.protocol.root (#2081) Review follow-ups on #2124 (@rzo1, @dpol1): - fileProtocolRejectsHostComponents now points at a real file inside the configured root: without the host check the request would resolve to that readable file and return 200, so the assertion (403) is tied to the host check specifically - the symlink escape test asserts 403 instead of not-200, which distinguishes the root-confinement rejection from the 300 the canonical-path redirect gives for a symlink inside the root - configuration.adoc: protocols now shows http,https as the shipped default and documents file.protocol.root Report URL: https://github.com/apache/stormcrawler/actions/runs/34383052454 With regards, GitHub Actions via GitBox
