Flink deployment resources support env interpolation natively using $() syntax. I expected this to "just work" like other resources when using the operator, but it does not.
https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments job: jarURI: local:///my.jar entryClass: my.JobMainKt args: - "--kafka.bootstrap.servers" - "my.kafka.host:9093" - "--kafka.sasl.username" - "$(KAFKA_SASL_USERNAME)" - "--kafka.sasl.password" - "$(KAFKA_SASL_PASSWORD)" It would be a great addition, simplifying job startup decision-making while following existing conventions. ________________________________ From: Yang Wang <danrtsey...@gmail.com> Sent: Tuesday, May 3, 2022 7:22 AM To: Őrhidi Mátyás <matyas.orh...@gmail.com> Cc: Francis Conroy <francis.con...@switchdin.com>; user <user@flink.apache.org> Subject: Re: Using the official flink operator and kubernetes secrets Flink could not support environment replacement in the args. I think you could access the env via "System.getenv()" in the user main method. It should work since the user main method is executed in the JobManager side. Best, Yang Őrhidi Mátyás <matyas.orh...@gmail.com<mailto:matyas.orh...@gmail.com>> 于2022年4月28日周四 19:27写道： Also, just declaring it in the flink configs should be sufficient, no need to define it in the pod templates: flinkConfiguration: kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" Best, Matyas On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <matyas.orh...@gmail.com<mailto:matyas.orh...@gmail.com>> wrote: Hi Francis, I suggest accessing the environment variables directly, no need to pass them as command arguments I guess. Best, Matyas On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy <francis.con...@switchdin.com<mailto:francis.con...@switchdin.com>> wrote: Hi all, I'm trying to use a kubernetes secret as a command line argument in my job and the text replacement doesn't seem to be happening. I've verified passing the custom args via the command line on my local flink cluster but can't seem to get the environment var replacement to work. apiVersion: flink.apache.org/v1alpha1<http://flink.apache.org/v1alpha1> kind: FlinkDeployment metadata: namespace: default name: http-over-mqtt spec: image: flink:1.14.4-scala_2.12-java11 flinkVersion: v1_14 flinkConfiguration: taskmanager.numberOfTaskSlots: "2" kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN" serviceAccount: flink jobManager: replicas: 1 resource: memory: "1024m" cpu: 1 taskManager: resource: memory: "1024m" cpu: 1 podTemplate: spec: serviceAccount: flink containers: - name: flink-main-container volumeMounts: - mountPath: /flink-job name: flink-jobs env: - name: DJANGO_TOKEN # kubectl create secret generic switchdin-django-token --from-literal=token='[TOKEN]' valueFrom: secretKeyRef: name: switchdin-django-token key: token optional: false initContainers: - name: grab-mqtt-over-http-jar image: docker-push.k8s.local/test/switchdin/platform_flink:job-41 command: [ "/bin/sh", "-c", "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in the init container to the flink-jobs volume volumeMounts: - name: flink-jobs mountPath: /tmp/job volumes: - name: flink-jobs emptyDir: { } job: jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar args: ["--swit-django-token", "$DJANGO_TOKEN", "--swit-prod","false"] entryClass: org.switchdin.HTTPOverMQTT parallelism: 1 upgradeMode: stateless state: running In the logs I can see: 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {} 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - --swit-django-token 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - $DJANGO_TOKEN 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false Anyone know how I can do this? I'm considering mounting it in a volume, but that seems like a lot of hassle for such a small thing. Thanks in advance! This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
