[
https://issues.apache.org/jira/browse/FLINK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamir Sagi updated FLINK-31140:
-------------------------------
Description:
To date is is not possible to add additional jars to operator classpath.
In our case, We have a custom layout which need to work with IAM authentication
and SSL against AWS MSK.
log4j.properties file
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
required;
appender.kafka.sasl.client.callback.handler.class =
software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.
main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
java.lang.NullPointerException java.lang.NullPointerException
furthermore, we need to add all the essential AWS dependencies.
To support additional jar we need to add them and then replace the following
lines
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
using 'sed' command
LOGGER_JAR in that case is an uber jar contains all the necessary jars.
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
RUN mkdir -p $USER_DEPENDENCIES_DIR
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
USER root
RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
/docker-entrypoint.sh
USER flink{code}
It works great but it should not be handled that way.
The idea of that ticket is to allow users to add additional jars into specific
location which is added to classpath while launching Kubernetes operator.
I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well
in root Dockerfile.
along with minor modification in docker-entrypoint.sh (Add
$USER_DEPENDENCIES_DIR to 'cp' command)
After that change, the only thing users have to do is copy all dependencies to
that folder ($USER_DEPENDENCIES_DIR)
I tested it locally and it seems to be working.
was:
To date is is not possible to add additional jars to operator classpath.
In our case, We have a custom layout which need to work with IAM authentication
and SSL against AWS MSK.
log4j.properties file
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
required;
appender.kafka.sasl.client.callback.handler.class =
software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.
main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
java.lang.NullPointerException java.lang.NullPointerException
furthermore, we need to add all the essential AWS dependencies.
To support additional jar we need to add them and then replace the following
lines
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
using 'sed' command
LOGGER_JAR in that case is an uber jar contains all the necessary jars.
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
USER root
RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
/docker-entrypoint.sh
USER flink{code}
It works great but it should not be handled that way.
The idea of that ticket is to allow users to add additional jars into specific
location which is added to classpath while launching Kubernetes operator.
I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well
in root Dockerfile.
along with minor modification in docker-entrypoint.sh (Add
$USER_DEPENDENCIES_DIR to 'cp' command)
After that change, the only thing users have to do is copy all dependencies to
that folder ($USER_DEPENDENCIES_DIR)
I tested it locally and it seems to be working.
> Load additional dependencies in Operator classpath
> ---------------------------------------------------
>
> Key: FLINK-31140
> URL: https://issues.apache.org/jira/browse/FLINK-31140
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Tamir Sagi
> Priority: Major
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> To date is is not possible to add additional jars to operator classpath.
> In our case, We have a custom layout which need to work with IAM
> authentication and SSL against AWS MSK.
> log4j.properties file
>
> {code:java}
> appender.kafka.type = Kafka
> appender.kafka.name = Kafka
> appender.kafka.bootstrap.servers = <brokers>
> appender.kafka.topic = <topic>
> appender.kafka.security.protocol = SASL_SSL
> appender.kafka.sasl.mechanism = AWS_MSK_IAM
> appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
> required;
> appender.kafka.sasl.client.callback.handler.class =
> software.amazon.msk.auth.iam.IAMClientCallbackHandler
> appender.kafka.layout.type = CustomJsonLayout
> appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
> appender.kafka.layout.service_name = <service-name>{code}
> if CustomLayout is not present in classpath the logger fails to start.
> main ERROR Could not create plugin of type class
> org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
> java.lang.NullPointerException java.lang.NullPointerException
> furthermore, we need to add all the essential AWS dependencies.
>
> To support additional jar we need to add them and then replace the following
> lines
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
> using 'sed' command
> LOGGER_JAR in that case is an uber jar contains all the necessary jars.
> {code:java}
> ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
> RUN mkdir -p $USER_DEPENDENCIES_DIR
> COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
> USER root
> RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
> /docker-entrypoint.sh
> USER flink{code}
> It works great but it should not be handled that way.
>
> The idea of that ticket is to allow users to add additional jars into
> specific location which is added to classpath while launching Kubernetes
> operator.
> I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as
> well in root Dockerfile.
> along with minor modification in docker-entrypoint.sh (Add
> $USER_DEPENDENCIES_DIR to 'cp' command)
>
> After that change, the only thing users have to do is copy all dependencies
> to that folder ($USER_DEPENDENCIES_DIR)
> I tested it locally and it seems to be working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
