wangzhigang1999 commented on PR #7451:
URL: https://github.com/apache/kyuubi/pull/7451#issuecomment-4953786127
> Thank you for testing this end-to-end. In my test environment,
`hive.metastore.uris` is defined in hive-site.xml, so I didn’t notice this
issue.
>
> My understanding is that among the native prefixes covered by
ConfigAudience, only `hive.*` may actually be required by the Spark SQL Engine.
>
> * Spark
>
> * Consumes `hive.*` configurations such as `hive.metastore.uris`, as you
pointed out.
> * Trino
>
> * Uses `trino.*` only to pass session properties to the engine and does
not reference other native prefixes.
> * Flink
>
> * Although it may use Hive configurations, Flink SQL specifies the
location of hive-site.xml through the catalog, so it does not rely on
forwarding `hive.*` configurations.
> * JDBC / Data Agent
>
> * Do not reference native prefixes.
>
> With that in mind, instead of treating all unregistered native
configurations as ANY, what do you think about introducing a
backward-compatible mechanism for selecting the engine audience, such as the
following configuration?
>
> ```scala
> val ENGINE_SPARK_EXTRA_CONF_PREFIXES: ConfigEntry[Seq[String]] =
> buildConf("kyuubi.config.engine.spark.extra.prefixes")
> .doc("A comma-separated list of config key prefixes that should be
forwarded to the " +
> "Spark engine in addition to spark.*, kyuubi.engine.spark.* and " +
> "kyuubi.session.engine.spark.* prefixes. " +
> "This is useful for native engine configs like hive.* that Spark
also consumes.")
> .audience(ConfigAudience.SERVER)
> .immutable
> .version("1.11.0")
> .stringConf
> .toSequence()
> .createWithDefault(Seq("hive."))
> ```
I still prefer keeping unregistered configurations as ANY and applying
audience filtering only to registered ConfigEntrys.
The extra-prefix configuration adds complexity and requires users to
maintain engine-specific allowlists. Defaulting it to hive. fixes this case,
but does not preserve compatibility: any existing unregistered prefix missing
from the list would be silently filtered after an upgrade.
Kyuubi knows the audience of registered entries, but cannot know how plugins
or downstream extensions consume unregistered configurations. Keeping those as
ANY is safer and preserves the current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
