andygrove opened a new pull request, #6344: URL: https://github.com/apache/datafusion-comet/pull/6344
## Which issue does this PR close? N/A. This is the ["Generate Release Documentation"](https://github.com/apache/datafusion-comet/blob/main/docs/source/contributor-guide/release_process.md#generate-release-documentation) step of the 1.1.0 release (#5327). ## Rationale for this change On `main`, `docs/source/user-guide/latest/` holds only template markers, and CI fills them at publish time. A release branch instead commits the generated content, so the archived 1.1.0 docs render real tables. ## What changes are included in this PR? Output of `./dev/generate-release-docs.sh`, with no hand edits: - `configs.md`: all 10 `CONFIG_TABLE` markers filled in (+446 lines). - `compatibility/expressions/spark-{3.4,3.5,4.0,4.1}/*.md`: 40 per-Spark-version compatibility pages generated from `_category_template/`. - `expressions.md`: Implementation column refreshed (16 rows, see below). A separate commit changes `dev/generate-release-docs.sh` to format with `npx --yes prettier@latest` instead of a bare `npx prettier`. `branch-1.0` carries the same change from #5244, but it never reached `main`. Without it, the script used the locally installed prettier 3.8.1, while CI installs the latest release (3.9.9). The two versions disagree in two ways: - 3.9.9 wants a blank line before `<!--END:EXPR_COMPAT-->` in 32 of the generated pages, so the lint job would fail. - 3.8.1 reformats the cast tables inside their `prettier-ignore` block and indents the `<!-- prettier-ignore-end -->` and `<!--END:CAST_*-->` markers into the last notes bullet. The 1.0.0 cast pages on `branch-1.0` have that damage. With 3.9.9 the block comes through exactly as GenerateDocs wrote it. ## Note for reviewers: `expressions.md` Implementation changes `docs/build.sh` runs the same four-profile loop, so the published `latest` page already shows these values. Freezing them just makes them visible in a diff. Seven rows now match the serdes: - `array_contains` (Native to Hybrid), `cast` (Native to Hybrid) and `unbase64` (Codegen dispatch to Hybrid): each serde mixes in `CodegenDispatchFallback`, and each row's note already describes both paths. - `to_unix_timestamp` (Hybrid to Codegen dispatch): #6182 removed the native opt-in. - `<<`, `>>` and `>>>` (placeholder to Native): Spark 4.0+ registers them as operator aliases for the shift functions. Nine rows now show the em-dash placeholder. They are generator limitations, not lost support: - `hour`, `minute`, `second`, `make_timestamp`, `mode`, `to_char` and `to_varchar`: Spark 4.1 registers these with `expressionBuilder(...)`, so `ExpressionInfo.getClassName` names the builder class, which has no serde. `spark-4.1` is the last profile in the loop and `expressions.md` is not per-version, so its answer wins. All but `mode` regressed the same way in 1.0.0 (#5244), and #6169 tracks the fix. - `regr_sxx` and `regr_syy`: Spark implements them as `RuntimeReplaceableAggregate`s that rewrite to `RegrReplacement` before Comet sees the plan. Comet's serde is keyed on `RegrReplacement`, so the generator finds nothing for `RegrSXX` or `RegrSYY`. They still run natively. #6169 does not list this case yet. If the archived 1.1.0 page should show these rows correctly, the fix belongs in `GenerateDocs` on `main`, backported here, and then this PR regenerated. ## How are these changes tested? - `./dev/generate-release-docs.sh` completed for all four Spark profiles on JDK 17. I ran it before and after the prettier change; the output differs only in the formatting described above. - `npx prettier@latest --check "**/*.md"` (3.9.9, the version CI installs) passes, and a `--write` pass changes nothing. - A Sphinx build of `docs/` shows no new warnings compared with `branch-1.1`, and 40 toctree warnings about the missing per-version pages go away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
