andygrove opened a new pull request, #5049:
URL: https://github.com/apache/datafusion-comet/pull/5049
## Which issue does this PR close?
<!-- No linked issue; opened at user request following review discussion on
#4852. -->
Supersedes #4852.
## Rationale for this change
There is currently no convenient way for a Spark user to query which Comet
version is loaded. The JVM `org.apache.comet.COMET_VERSION` constant exists but
is only reachable via Scala imports, and the native library logs the version on
init but that requires driver log access.
The previous attempt (#4852) exposed the version as a SQL function via
`SparkSessionExtensions.injectFunction`. Review feedback on that PR questioned
whether a function was the right mechanism, and registering a function into the
session registry also broke the upstream Spark `ShowFunctionsSuite` tests,
since those suites assert on the exact set of registered functions. Exposing
the version as a Spark config avoids polluting the function registry entirely
and matches how Spark surfaces similar build metadata.
## What changes are included in this PR?
- The Comet driver plugin sets `spark.comet.version` on the SparkConf during
`init`, so it is queryable at runtime via
`spark.conf.get("spark.comet.version")` or `SET spark.comet.version` in SQL. It
is set before the off-heap check so the version is reported even when Comet is
otherwise disabled.
- Documents both the new `spark.comet.version` runtime config and the
existing `org.apache.comet.COMET_VERSION` (plus `COMET_BRANCH` /
`COMET_REVISION`) programmatic accessors in the installation guide.
## How are these changes tested?
- New unit test in `CometPluginsSuite` (`Comet version is exposed as a Spark
config`) asserting `spark.comet.version` is visible on both the SparkContext
conf and the session runtime config, using the same plugin-loading harness that
covers the existing memory-overhead behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
