Hello beam community! I’m looking for some help solving an issue running a beam job on flink using --environment_type DOCKER.
I have a flink cluster running in kubernetes configured so the taskworkers can run docker [1]. I’m attempting to deploy a flink jobserver in the cluster. The issue is that the jobserver does not provide the proper endpoints to the SDK harness when it submits the job to flink. It typically provides something like localhost:34567 using the hostname the grpc server was bound to. There is a jobserver flag --job-host that will bind the grpc server to this provided hostname, but I cannot seem to get it to bind to the k8s jobservice Service name [2]. I’ve tried different flavors of FQDNs but haven’t had any luck. [main] INFO org.apache.beam.runners.jobsubmission.JobServerDriver - ArtifactStagingService started on flink-beam-jobserver:8098 [main] WARN org.apache.beam.runners.jobsubmission.JobServerDriver - Exception during job server creation java.io.IOException: Failed to bind ... Does anyone have some experience with this that could help provide some guidance? Cheers, Sam [1] https://github.com/sambvfx/beam-flink-k8s [2] https://github.com/sambvfx/beam-flink-k8s/blob/master/k8s/beam-flink-jobserver.yaml#L29
