Tamir Sagi created FLINK-31140:
----------------------------------
Summary: Load additional dependencies in Operator classpath
Key: FLINK-31140
URL: https://issues.apache.org/jira/browse/FLINK-31140
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Tamir Sagi
To date is is not possible to add additional jars to operator classpath.
In our case, We have a custom layout which need to work with IAM authentication
and SSL against AWS MSK.
log4j.properties file
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
required;
appender.kafka.sasl.client.callback.handler.class =
software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.
main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.KafkaAppender for element Console:
java.lang.NullPointerException java.lang.NullPointerException
furthermore, we need to add all the essential AWS dependencies.
To support additional jar we need to add them and then replace the following
lines
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
using 'sed' command
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
USER root
RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
/docker-entrypoint.sh
USER flink{code}
It works great but it should not be handled that way.
The idea of that ticket is to allow users to add additional jars into specific
location which is added to classpath while launching Kubernetes operator.
I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well
in root Dockerfile.
along with minor modification in docker-entrypoint.sh (Add
$USER_DEPENDENCIES_DIR to 'cp' command)
I tested it locally and it seems to be working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
