nick-boss-tech opened a new pull request, #5061: URL: https://github.com/apache/solr/pull/5061
🤖 *AI text below* 🤖 *(posted on behalf of Nick Shanin)* ## Problem [SOLR-18119](https://issues.apache.org/jira/browse/SOLR-18119): Changes.html generation is quietly skipped when `python3` is not found, and downstream tasks then fail confusingly. Hostetter's report on the Jira (2026-02-13) shows the silent skip and the resulting `checkSiteLinks`/`checkBrokenLinks` failures on the Windows Jenkins box. Three directions have been discussed on the Jira: - (a) Fail loudly when `python3` is missing: #4999. - (b) Fall back to GraalPy: #5034, by [~janhoy]. - (c) Replace the scripts with a JVM-native implementation. Smiley (2026-10-07): "Wouldn't it be simpler to convert these scripts to something JVM native like Java or Kotlin?" Hostetter (2026-10-07) ranked this "even better still", while noting the project should not wait for it if the other two patches are ready first; Smiley added on #4999 (2026-10-08): "I wonder if we want gradle to invoke Python at all." Jan added on the Jira (2026-10-08) that GraalPy may turn out to be "a stop-gap solution until someone are comfortable porting them", and: "If anyone fancy developing the kind of tooling we're talking about here in pure Java, then feel free." This PR is that port, offered for discussion alongside #4999 and #5034. It is not a claim that the direction is already decided, and it is not a request to displace the GraalPy work. ## Change The two Python scripts the documentation build shells out to are ported to Java in `build-tools/build-infra`, and the Gradle scripts call the Java tasks instead: - `ChangesToHtml`: port of `gradle/documentation/changes-to-html/changes2html.py`. The HTML template is taken from the script (lines 516-698). - `CheckJavadocLinks`: port of `dev-tools/scripts/checkJavadocLinks.py`, including an HTML tokenizer ported from CPython 3.12.12 `Lib/html/parser.py`. - `PythonCompat`: package-private helpers reproducing the Python behavior the scripts depend on (`str.strip` with Python's whitespace set, `re.escape`, `re.sub` with a function, `urllib.parse` quote/unquote/urlsplit/urlunsplit/urljoin, and a subset of `html.unescape`). - `ChangesToHtmlTask` and `CheckBrokenLinksTask`: `@CacheableTask` Gradle tasks, exposed through the `buildinfra` extension the same way `checksumClass()` already is. - `gradle/documentation/changes-to-html.gradle` and `gradle/validation/check-broken-links.gradle`: the inline Groovy task classes and the `python3` calls are removed; the tasks are registered from the `buildinfra` accessors. - `dev-docs/how-to-contribute.adoc`, `dev-docs/solr-source-code.adoc`: the `python3`-on-PATH note is removed. The two Python scripts are deleted in this PR (commit `8f7c1816fdc`). Nothing in the build references them after this change; the port was written and verified against them, and they remain in this branch's history at the commits before the deletion. Unit tests for the three classes (41 tests, added during a pre-submission review) pin the specified behavior, including the review-driven fixes in this branch: a file the HTML parser cannot process is reported as failed-to-parse and the crawl continues, matching the Python original's bare `except` (an earlier draft of the port aborted the crawl there); `Changes.html` and the `checkBrokenLinks` report are written with LF line endings on every platform, so cache keys and bytes do not depend on the build machine; numeric character references for the code points CPython treats as invalid decode the way `html.unescape` decodes them; and a missing docs directory fails fast with a message naming the directory. A changelog fragment (`changelog/unreleased/SOLR-18119.yml`, type `changed`) is included so the changelog check passes without a label. ## Proof Head `af7396db686` on `main` at `d94c5eeb6b5` (2026-10-08). Equivalence against the Python scripts (the scripts are the oracle; comparisons exclude exactly one line, the generated header comment, which now reads "ChangesToHtml, invoked by the changesToHtml task." instead of "Python script 'changes2html.py'."): - Real `CHANGELOG.md` (1.6 MB, 9,928 lines) converted both ways: identical except that header line. Six crafted changelogs (CRLF input, NBSP, zero-width space, U+2028, no releases, empty): same result. - A 28-file link fixture: 21 key lines and exit 1 from both implementations, matching the expected key file exactly. - A built site (8,213 HTML files): 0 key lines and exit 0 from both. With injected errors: 9 key lines and exit 1 from both. - Oracle comparisons were run on Windows against CPython 3.12.12 and on Linux against CPython 3.12.3 and 3.12.15. The fixture comparison matches literally under 3.12.12 and 3.12.15; 3.12.3 renders relative `file:` keys differently (see Limits). The entity code-point behavior was verified directly against CPython 3.12.15. Gradle wiring and tests, executed on Linux (Temurin JDK 21, Gradle 9.7.0) at the head above: - `:build-tools:build-infra:test`: 41 tests, 0 failures, 0 errors, 0 skipped (counted from the JUnit XML). - `:solr:documentation:changesToHtml` succeeds with no mention of Python in the log, at a head where the two scripts are already deleted from the tree. - The same task with `-Ppython3.exe=/does/not/exist` succeeds, and `Changes.html` is byte-identical (SHA-256 `457d199d1b7e41c661a37c79281234aa3c1d1dd129a1639162c7cadf5698b5f3`) to the normal run and to the standalone harness output. - `:solr:documentation:checkBrokenLinks` is green over a full documentation build (4m 18s, report has 0 key lines). - A repeat `changesToHtml` run reports UP-TO-DATE. - `spotlessCheck` passes, root `tidy` is clean, the changelog fragment parses, and `:solr:documentation:check -x test` is BUILD SUCCESSFUL. A Windows run of the full harness and Gradle steps at this head is being done separately; its results will be added here when they land. ## Choices 1. **Direction.** This PR implements the JVM-native option (c). If maintainers prefer keeping Python with the fail-fast of #4999 or the GraalPy fallback of #5034, this branch stands as a reference implementation and I will close it. I have not coordinated this with Jan (assignee, author of #5034) beyond the Jira thread. 2. **The Python scripts.** Implemented position: they are deleted in this PR, since the unit tests now pin the behavior they specified. If maintainers prefer keeping them in the tree as the specification, restoring them is a one-commit revert of `8f7c1816fdc`. 3. **Architecture.** This layout (plain Java classes in `build-infra`, thin `@CacheableTask` wrappers, accessors on the `buildinfra` extension) follows the existing `checksumClass()` pattern; it is not the only shape that would work. The same port could be written in Kotlin, as Smiley's comment suggested; the logic could live in a standalone tool project instead of `build-infra`; the Gradle scripts could keep small Groovy task classes that delegate to the plain Java converters instead of registering task classes through the extension; or GraalPy could be embedded through its Java API rather than run as a subprocess, which keeps Python source but removes the external interpreter. I picked the shape that reuses the build's existing conventions and keeps the converters free of Gradle types, so they can be compiled and tested with `javac` alone. Happy to restructure if maintainers prefer another shape. 4. **The helper class.** `PythonCompat` is one package-private class holding the string, URL and entity helpers both tools use. If maintainers prefer it renamed or split along its callers, I will do that in this PR. 5. **Anchor validation.** The checker does not validate fragment anchors for links that resolve (see Limits), in either implementation. If maintainers want anchor validation added, it fits as a follow-up change with its own site-wide measurement; I can also add it here if that is preferred. ## Limits - Fragment anchors are not validated for resolvable links: `other.html#missing-anchor` passes as long as `other.html` exists. The Python original has the same gap, and the port preserves it rather than silently changing what the checker promises. For links in schemes outside http, mailto, javascript and file, following the fragment into the anchor table errors in both implementations. - `html.unescape` is a subset: the named references `amp lt gt quot apos nbsp` (and legacy forms), and numeric references in 0x80-0x9F are not remapped to Windows-1252. Code points CPython treats as invalid are dropped, matching `html.unescape`. In the built site, the only entities appearing inside `href`, `name` or `id` attributes are `<` and `>`. - A missing docs directory fails fast with a message naming the directory, where Python's `os.walk` silently yielded nothing. Gradle always supplies the directory; the difference is deliberate. - `file:` key rendering depends on the Python version used as the comparison oracle (3.12.12 and 3.12.15 render one way, 3.12.3 another). The port follows 3.12.12. - This PR conflicts with #4999 and #5034 in `gradle/documentation/changes-to-html.gradle` and `gradle/validation/check-broken-links.gradle` if either merges first. 🤖 *This PR description was drafted with AI assistance on behalf of Nick Shanin.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
