Hello Michels, I am following below documentation and doing local testing only. Both job-server and flink cluster are on the same machine. The pipeline too is submitted from the same machine. I will drop the `--environment_type=LOOPBACK` flag and test again.
https://beam.apache.org/documentation/runners/flink/#dependencies -----Original Message----- From: Maximilian Michels <m...@apache.org> Sent: 01 June 2020 15:37 To: user@beam.apache.org Subject: Re: Issue while submitting python beam pipeline on flink - local EXTERNAL EMAIL Do not click links or open attachments unless you recognise the sender and know the content is safe. Report suspicious email to info...@corecompete.com. The logs indicate that you are not running the Docker-based execution but the `LOOPBACK` mode. In this mode the Flink cluster needs to connect to the machine that started the pipeline. That will not be possible unless you are running the Flink cluster on the same machine (we bind to `localhost` which prevents non-local connections). Please try the Docker-based execution, i.e. drop the `--environment_type=LOOPBACK` flag. On 31.05.20 20:25, Ashish Raghav wrote: > Hello Kyle, > > > > I did finally manage to run this setup on gcp after spinning up a > Debian server and doing the necessary setup to submit python beam > pipeline on flink job server and flink cluster ( I still cannot run > directly using FlinkRunner). > > The jobs are getting submitted on flink cluster , but they are failing > with these errors( attached logs). > > I have setup the -artifacts-dir as gs://<bucket>/prefix while running > the flink job server, but still it fails. I don't think this is access > issue as the instance on which the flink cluster is running has full > access to gcs. > > > > I tried following this > https://stackoverflow.com/questions/59429897/beam-running-on-flink-wit > h-python-sdk-and-using-google-cloud-storage-for-artifac > > But it seems, there are other things to do to fix this. > > > > Please suggest. > > > > > > *From:*Ashish Raghav <ashish.rag...@corecompete.com> > *Sent:* 30 May 2020 10:12 > *To:* user@beam.apache.org > *Subject:* Re: Issue while submitting python beam pipeline on flink - > local > > > > Hello Kyle, reply below. > > Also, what is the stack to run this as production setup on gcloud?? I > can try that setup to see if this works. > > > > Get Outlook for Android <https://aka.ms/ghei36> > > > > ---------------------------------------------------------------------- > -- > > *From:* Kyle Weaver <kcwea...@google.com <mailto:kcwea...@google.com>> > *Sent:* Thursday, May 28, 2020, 10:34 PM > *To:* user@beam.apache.org <mailto:user@beam.apache.org> > *Subject:* Re: Issue while submitting python beam pipeline on flink - > local > > > > *EXTERNAL EMAIL* > > Do not click links or open attachments unless you recognise the sender > and know the content is safe. Report suspicious email to > info...@corecompete.com <mailto:info...@corecompete.com>. > > > >> You are using the LOOPBACK environment which requires that the Flink >> cluster can connect back to your local machine. Since the loopback >> environment by defaults binds to localhost that should not be possible. > > > > On the Flink runner page, we recommend using --net=host to avoid the > kinds of networking issues Ashish is experiencing. But the Docker > documentation says the host networking option is only available on > Linux (https://docs.docker.com/network/host/). You will either have to: > > a) expose the required ports, or ==> I tried exposing 8099 and not > using --net=host on docker desktop for windows and it did not work. I > also tried running job server on gcloud shell and submitting pipeline > over gcloud shell after setting up all requirements. it fails with > same errors as local. > > b) run the job server in a local process instead of a Docker > container, as described in older versions of the documentation ==> I > havent tried it yet. > > (https://web.archive.org/web/20191228110311/https://beam.apache.org/do > cumentation/runners/flink/ > <https://web.archive.org/web/20191228110311/https:/beam.apache.org/doc > umentation/runners/flink/>) > > >
