[
https://issues.apache.org/jira/browse/FLINK-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503753#comment-17503753
]
Fil Karnicki edited comment on FLINK-26537 at 3/9/22, 5:30 PM:
---------------------------------------------------------------
hi [~igal], I think that depending on which classes get put on the classpath by
the existing cluster, you may or may not need to have this validation disabled.
I'm not sure there's a one-size-fits-all solution here, where for all embedded
runs we shouldn't check for these classloader settings. At the same time, for
people who give running the job on an existing cluster a go and face
classloading problems, we could let them disable this validation, provided that
they know that they need to include the correct version of statefun, kafka and
protobuf dependencies in the uber jar (and not for example protobuf 2.5.0 which
statefun 3.3 won't play nicely with).
was (Author: JIRAUSER284249):
hi [~igal], I think that depending on which classes get put on the classpath by
the existing cluster, you may or may not need to have this validation disabled.
I'm not sure there's a one-size-fits-all solution here, where for all embedded
runs we shouldn't check for these classloader settings. At the same time, for
people who give running the job on an existing cluster a go and face
classloading problems, we could let them disable this validation, provided that
they know that they need to include the correct version of statefun, kafka and
protobuf dependencies (and not for example protobuf 2.5.0 which statefun 3.3
won't play nicely with).
> Allow disabling StatefulFunctionsConfigValidator validation for
> classloader.parent-first-patterns.additional
> ------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-26537
> URL: https://issues.apache.org/jira/browse/FLINK-26537
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Reporter: Fil Karnicki
> Priority: Major
>
> For some deployments of stateful functions onto existing, shared clusters, it
> is impossible to tailor which classes exist on the classpath. An example
> would be a Cloudera Flink cluster, which adds protobuf-java classes that
> clash with statefun ones.
> Stateful functions require the following flink-config.yaml setting:
> {{classloader.parent-first-patterns.additional:
> org.apache.flink.statefun;org.apache.kafka;{+}*com.google.protobuf*{+}}}
> In the case of the cloudera flink cluster, this will cause old, 2.5.0
> protobuf classes to be loaded by statefun, which causes MethodNotFound
> exceptions.
> The idea is to allow disabling of the validation below, if some config
> setting is present in the global flink configuration, for example:
> statefun.validation.parent-first-classloader.disable=true
>
> {code:java}
> private static void validateParentFirstClassloaderPatterns(Configuration
> configuration) {
> final Set<String> parentFirstClassloaderPatterns =
> parentFirstClassloaderPatterns(configuration);
> if
> (!parentFirstClassloaderPatterns.containsAll(PARENT_FIRST_CLASSLOADER_PATTERNS))
> {
> throw new StatefulFunctionsInvalidConfigException(
> CoreOptions.ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL,
> "Must contain all of " + String.join(", ",
> PARENT_FIRST_CLASSLOADER_PATTERNS));
> }
> } {code}
>
> Then, we wouldn't need to contain com.google.protobuf in
> {{classloader.parent-first-patterns.additional:}} and it would be up to the
> statefun user to use protobuf classes which are compatible with their version
> of statefun.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
