[
https://issues.apache.org/jira/browse/FLINK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tamir Sagi updated FLINK-31140:
-------------------------------
Description:
To date is is not possible to add additional jars to operator classpath.
In our case, We have a custom layout which need to work with IAM authentication
and SSL against AWS MSK.
log4j.properties file
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
required;
appender.kafka.sasl.client.callback.handler.class =
software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.
main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
java.lang.NullPointerException java.lang.NullPointerException
furthermore, we need to add all the essential AWS dependencies.
To support additional jar we need to add them and then replace the following
lines
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
using 'sed' command
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
USER root
RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
/docker-entrypoint.sh
USER flink{code}
It works great but it should not be handled that way.
The idea of that ticket is to allow users to add additional jars into specific
location which is added to classpath while launching Kubernetes operator.
I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well
in root Dockerfile.
along with minor modification in docker-entrypoint.sh (Add
$USER_DEPENDENCIES_DIR to 'cp' command)
I tested it locally and it seems to be working.
was:
To date is is not possible to add additional jars to operator classpath.
In our case, We have a custom layout which need to work with IAM authentication
and SSL against AWS MSK.
log4j.properties file
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
required;
appender.kafka.sasl.client.callback.handler.class =
software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.
main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.KafkaAppender for element Console:
java.lang.NullPointerException java.lang.NullPointerException
furthermore, we need to add all the essential AWS dependencies.
To support additional jar we need to add them and then replace the following
lines
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
using 'sed' command
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
USER root
RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
/docker-entrypoint.sh
USER flink{code}
It works great but it should not be handled that way.
The idea of that ticket is to allow users to add additional jars into specific
location which is added to classpath while launching Kubernetes operator.
I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well
in root Dockerfile.
along with minor modification in docker-entrypoint.sh (Add
$USER_DEPENDENCIES_DIR to 'cp' command)
I tested it locally and it seems to be working.
> Load additional dependencies in Operator classpath
> ---------------------------------------------------
>
> Key: FLINK-31140
> URL: https://issues.apache.org/jira/browse/FLINK-31140
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
> Reporter: Tamir Sagi
> Priority: Major
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> To date is is not possible to add additional jars to operator classpath.
> In our case, We have a custom layout which need to work with IAM
> authentication and SSL against AWS MSK.
> log4j.properties file
>
> {code:java}
> appender.kafka.type = Kafka
> appender.kafka.name = Kafka
> appender.kafka.bootstrap.servers = <brokers>
> appender.kafka.topic = <topic>
> appender.kafka.security.protocol = SASL_SSL
> appender.kafka.sasl.mechanism = AWS_MSK_IAM
> appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule
> required;
> appender.kafka.sasl.client.callback.handler.class =
> software.amazon.msk.auth.iam.IAMClientCallbackHandler
> appender.kafka.layout.type = CustomJsonLayout
> appender.kafka.layout.class = our.package.layouts.CustomJsonLayout
> appender.kafka.layout.service_name = <service-name>{code}
> if CustomLayout is not present in classpath the logger fails to start.
> main ERROR Could not create plugin of type class
> org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka:
> java.lang.NullPointerException java.lang.NullPointerException
> furthermore, we need to add all the essential AWS dependencies.
>
> To support additional jar we need to add them and then replace the following
> lines
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
> using 'sed' command
>
>
> {code:java}
> ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
> COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR
> USER root
> RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/'
> /docker-entrypoint.sh
> USER flink{code}
>
> It works great but it should not be handled that way.
>
> The idea of that ticket is to allow users to add additional jars into
> specific location which is added to classpath while launching Kubernetes
> operator.
> I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as
> well in root Dockerfile.
> along with minor modification in docker-entrypoint.sh (Add
> $USER_DEPENDENCIES_DIR to 'cp' command)
>
> I tested it locally and it seems to be working.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
