Hi Red! Thank you for your interest and for checking out the official Flink Kubernetes Operator Project.
This project is still relatively new and so far our focus has been on running native Flink workloads (Flink Application/Session clusters, and session jobs) and we haven't tried running a Flink Beam job yet. I think in your case the best thing to try would be the application deployment <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/overview/#application-deployments>. The current assumption is that your Flink application workload is packaged into a fatjar with a main class that can be executed by the Flink jobmaster component. I am not sure to what extent this is possible with the current Beam implementation, it would be good to get input from the Beam community. I guess there are multiple ways to achieve this on a high level: - Bundle everything into a fatjar (runner + user jar + configs) - Have the runner and user jars separately in the docker image and configure flink to add them to the classpath Maybe someone already tried running Beam jobs using the Flink Application mode in Flink, that person should be able to provide a working answer :) Cheers, Gyula On Sat, May 21, 2022 at 8:04 AM Red Daly <redd...@gmail.com> wrote: > I created an example project with my attempts to get something working: > > https://github.com/gonzojive/beam-go-k8s > > The good news is I got something working with Beam, Go, Bazel, Flink, and > Kubernetes (minikube). The working version doesn't use the operator and > only runs in a single pod. > > I would still like to figure out how to use the Flink operator with a Go > Beam job. I'm not yet sure how the operator should work together with the > "beam_flink1.14_job_server" and the Go binary. > > On Fri, May 20, 2022 at 5:17 PM Red Daly <redd...@gmail.com> wrote: > >> Hi, >> >> This is a request for documentation to walk through using the Flink >> Kubernetes operator with the Go Beam SDK. I will plan to update the thread >> as I investigate. I hope this feedback is useful. >> >> A few observations: >> >> 1) The official "Quick Start guide >> <https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/>" >> for the Kubernetes operator is great. It would be great if there were a >> followup guide for how to use this operator to run a Beam job, especially a >> Beam Go job. >> >> 2) The closest to a walkthrough I have found is >> github.com/sambvfx/beam-flink-k8s, but that doesn't seem to use the >> operator at all. >> >> 3) There are 3 different Flink Kubernetes operators. (Lyft, >> GoogleCloudPlatform, and the Apache one.) The github repos for these don't >> have information about the other projects in their READMEs, from what I can >> tell. So it is a bit confusing. >> >> 4) There are a lot of concepts between Beam and Flink. To be a bit blunt, >> I don't want to learn many concepts up front. I just want to run a few >> commands and be up and running, then invest the time to learn more concepts >> later. >> >> - Red >> >
