Thanks Meissner Dylan for the suggestion. I have created a ticket [1] to track this requirement.
[1]. https://issues.apache.org/jira/browse/FLINK-27491 Best, Yang Francis Conroy <francis.con...@switchdin.com> 于2022年5月5日周四 06:06写道： > Hi all, > > Thanks for looking into this. Yeah, I kept trying different variations of > the replacement fields with no success. I'm trying to use the .getenv() > technique now but our cluster is having problems and I haven't been able to > reinstall the operator. > I'll reply once it's all working. > > Thanks, > Francis > > On Thu, 5 May 2022 at 03:23, Meissner, Dylan < > dylan.t.meiss...@nordstrom.com> wrote: > >> Flink deployment resources support env interpolation natively using $() >> syntax. I expected this to "just work" like other resources when using the >> operator, but it does not. >> >> >> https://kubernetes.io/docs/tasks/inject-data-application/_print/#use-environment-variables-to-define-arguments >> >> job: >> jarURI: local:///my.jar >> entryClass: my.JobMainKt >> args: >> - "--kafka.bootstrap.servers" >> - "my.kafka.host:9093" >> - "--kafka.sasl.username" >> - "$(KAFKA_SASL_USERNAME)" >> - "--kafka.sasl.password" >> - "$(KAFKA_SASL_PASSWORD)" >> >> >> It would be a great addition, simplifying job startup decision-making >> while following existing conventions. >> >> ------------------------------ >> *From:* Yang Wang <danrtsey...@gmail.com> >> *Sent:* Tuesday, May 3, 2022 7:22 AM >> *To:* Őrhidi Mátyás <matyas.orh...@gmail.com> >> *Cc:* Francis Conroy <francis.con...@switchdin.com>; user < >> user@flink.apache.org> >> *Subject:* Re: Using the official flink operator and kubernetes secrets >> >> Flink could not support environment replacement in the args. I think you >> could access the env via "*System.getenv()*" in the user main method. >> It should work since the user main method is executed in the JobManager >> side. >> >> Best, >> Yang >> >> Őrhidi Mátyás <matyas.orh...@gmail.com> 于2022年4月28日周四 19:27写道： >> >> Also, >> >> just declaring it in the flink configs should be sufficient, no need to >> define it in the pod templates: >> >> flinkConfiguration: >> kubernetes.env.secretKeyRef: >> "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" >> >> >> Best, >> Matyas >> >> On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás <matyas.orh...@gmail.com> >> wrote: >> >> Hi Francis, >> >> I suggest accessing the environment variables directly, no need to pass >> them as command arguments I guess. >> >> Best, >> Matyas >> >> On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy < >> francis.con...@switchdin.com> wrote: >> >> Hi all, >> >> I'm trying to use a kubernetes secret as a command line argument in my >> job and the text replacement doesn't seem to be happening. I've verified >> passing the custom args via the command line on my local flink cluster but >> can't seem to get the environment var replacement to work. >> >> apiVersion: flink.apache.org/v1alpha1 >> kind: FlinkDeployment >> metadata: >> namespace: default >> name: http-over-mqtt >> spec: >> image: flink:1.14.4-scala_2.12-java11 >> flinkVersion: v1_14 >> flinkConfiguration: >> taskmanager.numberOfTaskSlots: "2" >> kubernetes.env.secretKeyRef: >> "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" >> # containerized.taskmanager.env.DJANGO_TOKEN: "$DJANGO_TOKEN" >> serviceAccount: flink >> jobManager: >> replicas: 1 >> resource: >> memory: "1024m" >> cpu: 1 >> taskManager: >> resource: >> memory: "1024m" >> cpu: 1 >> podTemplate: >> spec: >> serviceAccount: flink >> containers: >> - name: flink-main-container >> volumeMounts: >> - mountPath: /flink-job >> name: flink-jobs >> env: >> - name: DJANGO_TOKEN # kubectl create secret generic >> switchdin-django-token --from-literal=token='[TOKEN]' >> valueFrom: >> secretKeyRef: >> name: switchdin-django-token >> key: token >> optional: false >> initContainers: >> - name: grab-mqtt-over-http-jar >> image: docker-push.k8s.local/test/switchdin/platform_flink:job-41 >> command: [ "/bin/sh", "-c", >> "cp /opt/switchdin/* /tmp/job/." ] # Copies the jar in >> the init container to the flink-jobs volume >> volumeMounts: >> - name: flink-jobs >> mountPath: /tmp/job >> volumes: >> - name: flink-jobs >> emptyDir: { } >> job: >> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar >> args: ["--swit-django-token", "$DJANGO_TOKEN", >> "--swit-prod","false"] >> entryClass: org.switchdin.HTTPOverMQTT >> parallelism: 1 >> upgradeMode: stateless >> state: running >> >> In the logs I can see: >> >> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - ARGS ARE {} >> 2022-04-28 08:43:02,329 WARN org.switchdin.HTTPOverMQTT [] - >> --swit-django-token >> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - >> $DJANGO_TOKEN >> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - --swit-prod >> 2022-04-28 08:43:02,330 WARN org.switchdin.HTTPOverMQTT [] - false >> >> Anyone know how I can do this? I'm considering mounting it in a volume, >> but that seems like a lot of hassle for such a small thing. >> >> Thanks in advance! >> >> >> This email and any attachments are proprietary and confidential and are >> intended solely for the use of the individual to whom it is addressed. Any >> views or opinions expressed are solely those of the author and do not >> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have >> received this email in error, please let us know immediately by reply email >> and delete it from your system. You may not use, disseminate, distribute or >> copy this message nor disclose its contents to anyone. >> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 >> Australia >> >> > This email and any attachments are proprietary and confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily reflect or represent those of SwitchDin Pty Ltd. If you have > received this email in error, please let us know immediately by reply email > and delete it from your system. You may not use, disseminate, distribute or > copy this message nor disclose its contents to anyone. > SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 > Australia >
