Hello beam community!

I’m looking for some help solving an issue running a beam job on flink
using --environment_type DOCKER.

I have a flink cluster running in kubernetes configured so the taskworkers
can run docker [1]. I’m attempting to deploy a flink jobserver in the
cluster. The issue is that the jobserver does not provide the proper
endpoints to the SDK harness when it submits the job to flink. It typically
provides something like localhost:34567 using the hostname the grpc server
was bound to. There is a jobserver flag --job-host that will bind the grpc
server to this provided hostname, but I cannot seem to get it to bind to
the k8s jobservice Service name [2]. I’ve tried different flavors of FQDNs
but haven’t had any luck.

[main] INFO org.apache.beam.runners.jobsubmission.JobServerDriver -
ArtifactStagingService started on flink-beam-jobserver:8098
[main] WARN org.apache.beam.runners.jobsubmission.JobServerDriver -
Exception during job server creation
java.io.IOException: Failed to bind
...

Does anyone have some experience with this that could help provide some
guidance?

Cheers,
Sam

[1] https://github.com/sambvfx/beam-flink-k8s
[2]
https://github.com/sambvfx/beam-flink-k8s/blob/master/k8s/beam-flink-jobserver.yaml#L29

Reply via email to