jarredhj0214 commented on PR #13119: URL: https://github.com/apache/gravitino/pull/13119#issuecomment-5690728542
> I think adding this configuration option is a bit odd. It feels more like a workaround than a proper solution. @jerryshao @yuqi1129 What do you think Thanks for the feedback. I agree that if this were only for one specific incompatibility, it would look like a workaround. But I think the problem is more general. The Spark connector is usually enabled through `spark.plugins`, and in many platforms this configuration is injected by a shared job template or cluster-level policy. Once it is injected globally, every Spark application on that platform will load the connector, including existing jobs and jobs using third-party Spark components. The connector cannot realistically guarantee compatibility with every Spark DataSourceV2 implementation, Spark extension, or third-party planner behavior. TiSpark is just one example. There may be other existing workloads that are not related to Gravitino at all, but are still affected because the plugin is globally configured. So the goal of this option is not to hide a specific bug. It provides a job-level opt-out switch for incremental rollout and compatibility isolation: - Platform owners can keep Gravitino enabled by default. - Existing or incompatible jobs can explicitly opt out without changing the global template. - The default value remains `true`, so current behavior is unchanged. - The switch only disables Gravitino initialization and Gravitino extension behavior for that Spark application. This is similar to other feature flags used during gradual adoption. Without such a switch, the only workaround is to remove `spark.plugins` or `spark.sql.extensions` from the shared template, which disables Gravitino for all jobs and makes incremental adoption much harder. I think this is useful as a general safety mechanism for production platforms, not only for TiSpark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
