Hi all,
thanks in advance for any tips.
I've been trying to specify some additional classpaths in my kubernetes
yaml file when using the official flink operator and nothing seems to work.
I know the technique for getting my job jar works fine since it's finding
the class ok, but I cannot get the RabbitMQ connector jar to load.
apiVersion: flink.apache.org/v1alpha1
kind: FlinkDeployment
metadata:
namespace: default
name: http-over-mqtt
spec:
image: flink:1.14.4-scala_2.12-java11
flinkVersion: v1_14
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
pipeline.classpaths:
"file:///flink-jar/flink-connector-rabbitmq_2.12-1.14.4.jar"
serviceAccount: flink
jobManager:
replicas: 1
resource:
memory: "1024m"
cpu: 1
taskManager:
resource:
memory: "1024m"
cpu: 1
podTemplate:
spec:
serviceAccount: flink
containers:
- name: flink-main-container
volumeMounts:
- mountPath: /flink-job
name: flink-jobs
- mountPath: /flink-jars
name: flink-jars
initContainers:
- name: grab-mqtt-over-http-jar
image: busybox
command: [ '/bin/sh', '-c',
'cd /tmp/job; wget
https://jenkins/job/platform_flink/job/master/39/artifact/src-java/switchdin-topologies/target/switchdin-topologies-1.0-SNAPSHOT.jar
--no-check-certificate;',
'cd /tmp/jar; wget
https://repo1.maven.org/maven2/org/apache/flink/flink-connector-rabbitmq_2.12/1.14.4/flink-connector-rabbitmq_2.12-1.14.4.jar'
]
volumeMounts:
- name: flink-jobs
mountPath: /tmp/job
- name: flink-jars
mountPath: /tmp/jar
volumes:
- name: flink-jobs
emptyDir: { }
- name: flink-jars
emptyDir: { }
job:
jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar
entryClass: org.switchdin.HTTPOverMQTT
parallelism: 1
upgradeMode: stateless
state: running
Any ideas? I've looked at the ConfigMaps that result and they also look
fine.
apiVersion: v1
data:
flink-conf.yaml: "blob.server.port: 6124\nkubernetes.jobmanager.replicas:
1\njobmanager.rpc.address:
http-over-mqtt.default\nkubernetes.taskmanager.cpu: 1.0\n
kubernetes.service-account:
flink\nkubernetes.cluster-id: http-over-mqtt\n
$internal.application.program-args:
\nkubernetes.container.image: flink:1.14.4-scala_2.12-java11\n
parallelism.default:
1\nkubernetes.namespace: default\ntaskmanager.numberOfTaskSlots: 2\n
kubernetes.rest-service.exposed.type:
ClusterIP\n$internal.application.main: org.switchdin.HTTPOverMQTT\n
taskmanager.memory.process.size:
1024m\nkubernetes.internal.jobmanager.entrypoint.class:
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
\nkubernetes.pod-template-file:
/tmp/podTemplate_11292791104169595925.yaml\n
kubernetes.pod-template-file.taskmanager:
/tmp/podTemplate_17362225267763549900.yaml\nexecution.target:
kubernetes-application\njobmanager.memory.process.size:
1024m\njobmanager.rpc.port: 6123\ntaskmanager.rpc.port: 6122\n
internal.cluster.execution-mode:
NORMAL\nqueryable-state.proxy.ports: 6125\npipeline.jars:
local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar\n
kubernetes.jobmanager.cpu:
1.0\npipeline.classpath:
file:///flink-jars/flink-connector-rabbitmq_2.12-1.14.4.jar\n
kubernetes.pod-template-file.jobmanager:
/tmp/podTemplate_17029501154997462433.yaml\n"
--
This email and any attachments are proprietary and confidential and are
intended solely for the use of the individual to whom it is addressed. Any
views or opinions expressed are solely those of the author and do not
necessarily reflect or represent those of SwitchDin Pty Ltd. If you have
received this email in error, please let us know immediately by reply email
and delete it from your system. You may not use, disseminate, distribute or
copy this message nor disclose its contents to anyone.
SwitchDin Pty Ltd
(ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
