jnioche opened a new pull request, #2258: URL: https://github.com/apache/stormcrawler/pull/2258
Follow-up to the review comment https://github.com/apache/stormcrawler/pull/2256#discussion_r4359201671, kept separate from #2256. ## Problem Both XPathFilter implementations in `core` keep the first expression that matches for a key: - `org.apache.stormcrawler.jsoup.XPathFilter`, used by the JSoupFilters - `org.apache.stormcrawler.parse.filter.XPathFilter`, used by the ParseFilters They keep it even when its value is empty or whitespace-only. With the usual configuration for `parse.title` (`//TITLE` first, then a META fallback), a page with an empty `<title>` gets an empty `parse.title`, and the fallback never runs. This already happens with HTML pages. It will be more common once #2256 runs the JSoupFilters on documents parsed by Tika, because Tika always writes a `<title>` element, even for an untitled PDF. ## Change - Blank values are dropped. If an expression gives only blank values, the next expression for that key is tried. - If every expression for a key gives a blank value, nothing is stored for that key. ## Behaviour change Blank values are no longer stored at all, even next to non-blank ones. For example, `//SCRIPT[@type='text/javascript']` on the test page used to return two values: an empty one for the external `<script src=...>` and the inline script. It now returns only the inline script. `testScriptExtraction` in both test classes is updated to match. ## Tests New tests in `JSoupFiltersTest` and `XPathFilterTest`: - A page with an empty `<title>` and a `<meta name="title">` fallback stores the fallback. - A page where every expression gives a blank value stores nothing. `mvn -pl core test -Dtest='XPathFilterTest,JSoupFiltersTest'`: 18 tests, all pass. Without the change to the filters, the new tests fail. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
