Hi Robert,
I replied to your main question on SO. (This is becoming a frequently asked
question, so we're going to look for ways to improve or at least document
this process. Stay tuned.)
> Generally, I’d like to have some sort of state diagram to describe who
calls what when, if anything like that is available.
The command specified in environment_config should be able to handle any
work Beam sends its way, so I don't think such a feature would be
necessary. But then again, I'm not totally sure what you mean here?
On Thu, Nov 7, 2019 at 10:15 AM Robert Lugg <robert.l...@synopsys.com>
wrote:
> Hi All,
>
>
>
> I have a conceptual misunderstanding which is keeping me from running BEAM
> using Flink. The main killer right now is that I cannot use Docker
> containers. I would like to run using something like:
>
>
>
> python -m wordcount_flink \
>
> --input war_and_peace.txt \
>
> --runner=PortableRunner \
>
> --job_endpoint=my-host:8099 \
>
> --output result_flink
>
>
>
> The first thing I did was start up a Flink cluster using
> ${FLINK_ROOT}/bin/start-cluster.sh. The website confirms I have workers
> running.
>
>
>
> Then, I started up a JobServer:
>
> ./gradlew :runners:flink:1.8:job-server:*runShadow* \
>
> -PflinkMasterUrl=my-host:8082
>
>
>
> Finally, I attempted to run the wordcount sample:
>
> python -m wordcount_flink \
>
> --input war_and_peace.txt \
>
> --runner=PortableRunner \
>
> --job_endpoint=my-host:8099 \
>
> --output result_flink
>
>
>
> This failed with:
>
> … java.io.IOException: Received exit code 126 for command 'docker run -d
> --network=host --env=DOCKER_MAC_CONTAINER=null
> apachebeam/python3.6_sdk:2.16.0 --id=1-1 --logging_endpoint=localhost:39858
> --artifact_endpoint=localhost:39422 --provision_endpoint=localhost:40678
> --control_endpoint=localhost:35088'
>
> … docker: Got permission denied while trying to connect to the Docker
> daemon socket at unix:///var/run/docker.soc
>
>
>
> I want to instead use the FlinkRunner and environment_type=PROCESS and
> environment_conf=???? What do I need to do to do this?
>
>
>
> Generally, I’d like to have some sort of state diagram to describe who
> calls what when, if anything like that is available.
>
>
>
> Thank you for any guidance.
>
>
>
>
>
>
>
>
>
>
>
> Please note: I have cross posted this on stackoverflow:
> https://stackoverflow.com/questions/58718098/what-environment-config-for-beam-launching-flink
>
> If I get an answer here, I’ll enter it there (or you can).
>
>
>
> Regards.
>
