Hi John, No such deployment or services in the K8S cluster. Same issue happens to the flink native kubernetes deployment. We have the podsecuritypolicy defined, but I have added flink service account into the psp.
*Xiao Ma* *Geotab* Software Developer, Data Engineering | B.Sc, M.Sc Direct +1 (416) 836 - 3541 Toll-free +1 (877) 436 - 8221 Visit www.geotab.com Twitter <https://twitter.com/geotab> | Facebook <https://www.facebook.com/Geotab> | YouTube <https://www.youtube.com/user/MyGeotab> | LinkedIn <https://www.linkedin.com/company/geotab/> On Tue, May 17, 2022 at 9:50 PM John Gerassimou <john.gerassi...@unity3d.com> wrote: > Hi Xiao, > > Is istio or something similar deployed to the K8S cluster? > > John > > On Tue, May 17, 2022 at 4:26 PM Xiao Ma <xia...@geotab.com> wrote: > >> loop in >> *Xiao Ma* >> *Geotab* >> Software Developer, Data Engineering | B.Sc, M.Sc >> Direct +1 (416) 836 - 3541 >> Toll-free +1 (877) 436 - 8221 >> Visit www.geotab.com >> Twitter <https://twitter.com/geotab> | Facebook >> <https://www.facebook.com/Geotab> | YouTube >> <https://www.youtube.com/user/MyGeotab> | LinkedIn >> <https://www.linkedin.com/company/geotab/> >> >> >> ---------- Forwarded message --------- >> From: Xiao Ma <xia...@geotab.com> >> Date: Tue, May 17, 2022 at 4:18 PM >> Subject: Re: Question of Flink Operator Application Cluster Deployment >> To: Őrhidi Mátyás <matyas.orh...@gmail.com> >> >> >> Fyi, I didn't manually mount the service account token into the job pod. >> It is automatically mounted into the pod, with the "bound service account >> token volume". I also found that the fabric8 cannot read the service >> account token if it is the "bound service account token volume". link: >> https://github.com/fabric8io/kubernetes-client/issues/2271 >> >> Thank you very much. >> >> Best, >> *Xiao Ma* >> *Geotab* >> Software Developer, Data Engineering | B.Sc, M.Sc >> Direct +1 (416) 836 - 3541 >> Toll-free +1 (877) 436 - 8221 >> Visit www.geotab.com >> Twitter <https://twitter.com/geotab> | Facebook >> <https://www.facebook.com/Geotab> | YouTube >> <https://www.youtube.com/user/MyGeotab> | LinkedIn >> <https://www.linkedin.com/company/geotab/> >> >> >> On Tue, May 17, 2022 at 10:55 AM Xiao Ma <xia...@geotab.com> wrote: >> >>> Hi Őrhidi, >>> >>> Thank you very much for the help. >>> >>> The attached are flink-operator yaml files and the application job yaml >>> file. >>> >>> Best, >>> *Xiao Ma* >>> *Geotab* >>> Software Developer, Data Engineering | B.Sc, M.Sc >>> Direct +1 (416) 836 - 3541 >>> Toll-free +1 (877) 436 - 8221 >>> Visit www.geotab.com >>> Twitter <https://twitter.com/geotab> | Facebook >>> <https://www.facebook.com/Geotab> | YouTube >>> <https://www.youtube.com/user/MyGeotab> | LinkedIn >>> <https://www.linkedin.com/company/geotab/> >>> >>> >>> On Tue, May 17, 2022 at 12:22 AM Őrhidi Mátyás <matyas.orh...@gmail.com> >>> wrote: >>> >>>> You don't have to mount the service account explicitly, this should >>>> be auto-mounted for you. Please share your (redacted) yamls for the RBAC >>>> configs ( >>>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/#cluster-scoped-flink-operator-with-jobs-running-in-other-namespaces) >>>> and your deployment yaml, we could probably spot what's missing. >>>> >>>> Best, >>>> Matyas >>>> >>>> On Tue, May 17, 2022 at 5:37 AM Xiao Ma <xia...@geotab.com> wrote: >>>> >>>>> Hi Flink Community, >>>>> >>>>> First of all, I would like to express my great thankfulness about the >>>>> flink operator on Kubernetes. It is a new door to help us deploy the Flink >>>>> application on top of the K8s. >>>>> >>>>> Our team is currently doing the Application cluster deployment through >>>>> the operator. We have set up the service account as "flink-operator" and >>>>> "flink", with the roles and rolebindings. However, after the job yaml is >>>>> submitted to the api-server and the pod is created, the resources manager >>>>> cannot be created because this error log: >>>>> ==== >>>>> 2022-05-17 02:37:22,293 WARN io.fabric8.kubernetes.client.Config >>>>> [] - Error reading service account token from: >>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. >>>>> 2022-05-17 02:37:22,308 WARN io.fabric8.kubernetes.client.Config >>>>> [] - Error reading service account token from: >>>>> [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring. >>>>> 2022-05-17 02:37:25,699 INFO >>>>> org.apache.flink.runtime.jobmaster.JobMaster [] - >>>>> Connecting to ResourceManager akka.tcp://fl...@flink-application-job.bip >>>>> :6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) >>>>> 2022-05-17 02:37:26,094 WARN >>>>> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener [] - >>>>> Exec Failure: HTTP 403, Status: 403 - pods is forbidden: User >>>>> "system:anonymous" cannot watch resource "pods" in API group "" in the >>>>> namespace "xxxxxxxxx" >>>>> ==== >>>>> >>>>> It looks like the jobmanager pod cannot fetch the "flink" service >>>>> account token and cannot communicate with api-server, though I have >>>>> created >>>>> the "flink" service account and set up "serviceAccount" config in the job >>>>> template. >>>>> ==== >>>>> >>>>> apiVersion: flink.apache.org/v1beta1 >>>>> kind: FlinkDeployment >>>>> metadata: >>>>> name: flink-application-job >>>>> spec: >>>>> image: flink:1.15.0-scala_2.12-java11 >>>>> flinkVersion: v1_15 >>>>> flinkConfiguration: >>>>> taskmanager.numberOfTaskSlots: "2" >>>>> jobmanager.rpc.address: flink-jobmanager >>>>> serviceAccount: flink >>>>> >>>>> ==== >>>>> >>>>> The below shows the volumeMounts in the pod. The service account is >>>>> mounted through the "bound service account token volume". Is it desirable? >>>>> ==== >>>>> Mounts: >>>>> /opt/flink/conf from flink-config-volume (rw) >>>>> /opt/flink/log from flink-logs (rw) >>>>> /opt/flink/pod-template from pod-template-volume (rw) >>>>> /var/run/secrets/kubernetes.io/serviceaccount from >>>>> kube-api-access-f69zl (ro) >>>>> ==== >>>>> >>>>> This issue has blocked our progress for several days so if there are >>>>> any possible thoughts, we really appreciate it! >>>>> >>>>> Thank you very much and I'm looking forward to your reply. >>>>> >>>>> >>>>> Best, >>>>> *Xiao Ma* >>>>> *Geotab* >>>>> Software Developer, Data Engineering | B.Sc, M.Sc >>>>> Direct +1 (416) 836 - 3541 >>>>> Toll-free +1 (877) 436 - 8221 >>>>> Visit www.geotab.com >>>>> Twitter <https://twitter.com/geotab> | Facebook >>>>> <https://www.facebook.com/Geotab> | YouTube >>>>> <https://www.youtube.com/user/MyGeotab> | LinkedIn >>>>> <https://www.linkedin.com/company/geotab/> >>>>> >>>>
