[
https://issues.apache.org/jira/browse/FLINK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamir Sagi updated FLINK-31140:
-------------------------------
Attachment: image-2023-02-20-15-29-08-678.png
> Load additional dependencies in operator classpath
> ---------------------------------------------------
>
> Key: FLINK-31140
> URL: https://issues.apache.org/jira/browse/FLINK-31140
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Tamir Sagi
> Priority: Major
> Attachments: image-2023-02-20-15-29-08-678.png
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> To date is is not possible to add additional jars to operator classpath.
> In our case, We have a Kafka appender with custom layout that works with IAM
> authentication and SSL along with AWS MSK.
> log4j.properties file
>
> {code:java}
> appender.kafka.type = Kafka
> appender.kafka.name = Kafka
> appender.kafka.bootstrap.servers = <brokers>
> appender.kafka.topic = <topic>
> appender.kafka.security.protocol = SASL_SSL
> appender.kafka.sasl.mechanism = AWS_MSK_IAM
> appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
> required;
> appender.kafka.sasl.client.callback.handler.class =
> software.amazon.msk.auth.iam.IAMClientCallbackHandler
> appender.kafka.layout.type = CustomJsonLayout
> appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
> appender.kafka.layout.service_name = <service-name>{code}
> if CustomLayout is not present in classpath the logger fails to start.
> *main ERROR Could not create plugin of type class
> org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
> java.lang.NullPointerException java.lang.NullPointerException*
> furthermore, all the essential AWS dependencies must be added to path.
>
> To support additional jars we need to add them and then replace the following
> lines
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
> using 'sed' command
> LOGGER_JAR in that case is an uber jar contains all the necessary jars.
> {code:java}
> ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
> RUN mkdir -p $USER_DEPENDENCIES_DIR
> COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
> USER root
> RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
> /docker-entrypoint.sh
> USER flink{code}
> It works great but it should not be handled that way.
>
> The idea of that ticket is to allow users to add additional jars into
> specific location which is added to classpath while Kubernetes operator
> starts.
> I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as
> well in root Dockerfile.
> along with a minor modification in 'docker-entrypoint.sh' (Add
> $USER_DEPENDENCIES_DIR to 'cp' command)
>
> Once it's supported , the only thing users have to do is copy all
> dependencies to that folder ($USER_DEPENDENCIES_DIR) while building custom
> image
> I tested it locally and it seems to be working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
