Hi Apache Beam team,
I hope this email finds you all well. I have been experimenting with Apache
Beam and Flink, mainly using golang. I hit a roadblock when trying to run
the minimal word count example on Beam and Flink locally using Go SDK
workers. I am trying to use the "apache/beam_go_sdk:2.46.0" Docker image as
follows:
docker run --network=host apache/beam_go_sdk:2.46.0
--id=1-1 --provision_endpoint=localhost:50000 <-- (I set this port based
on some research online, but I don't really know what the service should be)
However, I am unable to understand what the following options represent:
Usage of /opt/apache/beam/boot:
-artifact_endpoint string
Local artifact endpoint for FnHarness (required).
-control_endpoint string
Local control endpoint for FnHarness (required).
-id string
Local identifier (required).
-logging_endpoint string
Local logging endpoint for FnHarness (required).
-provision_endpoint string
Local provision endpoint for FnHarness (required).
-semi_persist_dir string
Local semi-persistent directory (optional). (default "/tmp")
I checked:
https://github.com/apache/beam/blob/master/sdks/go/container/boot.go but
still unable to tell what these endpoints are. I couldn't find any online
documentation describing, for example, what the provision_endpoint should
be set to.
I would greatly appreciate any pointers or explanation.
My setup is as follows: I have a Flink JobManager, two TaskManagers, and a
Beam JobServer running locally. I can execute the pipeline that's written
in Go and see the job submitted on Flink's UI, however, it quickly fails
because there are no workers to execute the Go transforms.
Thanks,
Sherif Tolba
