Kevin Sijo Puthusseri created BEAM-10793:
--------------------------------------------
Summary: Incorrect Flink runner documentation
Key: BEAM-10793
URL: https://issues.apache.org/jira/browse/BEAM-10793
Project: Beam
Issue Type: Bug
Components: runner-flink, sdk-go
Reporter: Kevin Sijo Puthusseri
As per the documentation at
[https://beam.apache.org/documentation/runners/flink/] under _"Portable
(Java/Python/Go)"_, a containerized flink job server needs to be started using
{code:java}
docker run --net=host apache/beam_flink1.10_job_server:latest
{code}
or
{code:java}
docker run --net=host apache/beam_flink1.10_job_server:latest
--flink-master=localhost:8081
{code}
If any of the SDKs are run using the DOCKER flag, all crash. As explained by
[~danoliveira] – _"This command is building and running it locally on your
machine. I'm not 100% sure why running it in a container is causing the error,
but my suspicion is that it has to do with writing the manifest/artifact files
to disk. One thing the job server does is writing artifacts to disk and then
sending the locations to the SDK harness so it can read them. If the job server
is in a container, then its probably writing the files to the container instead
of your local machine, so they're inaccessible to the SDK harness."_ In fact,
[~lostluck] tracked this to an already existing issue
https://issues.apache.org/jira/browse/BEAM-5273 which is yet to be resolved and
addresses this exact problem. Using Daniel's advice, Go SDK (and others I'm
certain) can be run in DOCKER mode if the flink job server is started locally
using gradle as follows –
{code:java}
./gradlew :runners:flink:1.10:job-server:runShadow -Djob-host=localhost
-Dflink-master=local{code}
Only if the SDK is run using the LOOPBACK flag does it manage to run on a
containerized flink cluster. Moreoever since the LOOPBACK flag is explicitly
meant for *local development* purposes only, this makes me wonder how folks are
deploying their production beam data pipelines on flink (especially on managed
services like Kubernetes). Overall, the main issue (at least until BEAM-5273 is
unresolved) is the fact that beam's documentation fails to mention these
caveats explicitly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
