Re: Getting Beam(Python)-on-Flink-on-k8s to work

On Sat, Aug 29, 2020 at 10:59 AM Eugene Kirpichov wrote: > > On Fri, Aug 28, 2020 at 6:52 PM Sam Bourne wrote: > >> Hi Eugene, >> >> Glad that helped you out and thanks for the PR tweaking it for GCP. >> >> To fetch the containers from GCR, I had to log into Docker inside the >> Flink nodes, sp

Re: Getting Beam(Python)-on-Flink-on-k8s to work

On Fri, Aug 28, 2020 at 6:52 PM Sam Bourne wrote: > Hi Eugene, > > Glad that helped you out and thanks for the PR tweaking it for GCP. > > To fetch the containers from GCR, I had to log into Docker inside the > Flink nodes, specifically inside the taskmanager container, using something > like “ku

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Eugene, Glad that helped you out and thanks for the PR tweaking it for GCP. To fetch the containers from GCR, I had to log into Docker inside the Flink nodes, specifically inside the taskmanager container, using something like “kubectl exec pod/flink-taskmanager-blahblah -c taskmanager — docke

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Woohoo thanks Kyle, adding --save_main_session made it work!!! On Fri, Aug 28, 2020 at 5:02 PM Kyle Weaver wrote: > > rpc error: code = Unimplemented desc = Method not found: > org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService/GetManifest > > This is a known issue: https://i

Re: Getting Beam(Python)-on-Flink-on-k8s to work

> rpc error: code = Unimplemented desc = Method not found: org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService/GetManifest This is a known issue: https://issues.apache.org/jira/browse/BEAM-10762 On Fri, Aug 28, 2020 at 4:57 PM Eugene Kirpichov wrote: > P.S. Ironic how back in

Re: Getting Beam(Python)-on-Flink-on-k8s to work

P.S. Ironic how back in 2018 I was TL-ing the portable runners effort for a few months on Google side, and now I need community help to get it to work at all. Still pretty miraculous how far Beam's portability has come since then, even if it has a steep learning curve. On Fri, Aug 28, 2020 at 4:54

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Sam, You're a wizard - this got me *way* farther than my previous attempts. Here's a PR https://github.com/sambvfx/beam-flink-k8s/pull/1 with a couple of changes I had to make. I had to make some additional changes that do not make sense to share, but here they are for the record: - Because I'

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Holy shit, thanks Sam, this is more help than I could have asked for!! I'll give this a shot later today and report back. On Thu, Aug 27, 2020 at 10:27 PM Sam Bourne wrote: > Hi Eugene! > > I’m struggling to find complete documentation on how to do this. There > seems to be lots of conflicting o

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Eugene! I’m struggling to find complete documentation on how to do this. There seems to be lots of conflicting or incomplete information: several ways to deploy Flink, several ways to get Beam working with it, bizarre StackOverflow questions, and no documentation explaining a complete working e

Re: Getting Beam(Python)-on-Flink-on-k8s to work

Hi Kyle, Thanks for the response! On Wed, Aug 26, 2020 at 5:28 PM Kyle Weaver wrote: > > - With the Flink operator, I was able to submit a Beam job, but hit the > issue that I need Docker installed on my Flink nodes. I haven't yet tried > changing the operator's yaml files to add Docker inside

Re: Getting Beam(Python)-on-Flink-on-k8s to work

> - With the Flink operator, I was able to submit a Beam job, but hit the issue that I need Docker installed on my Flink nodes. I haven't yet tried changing the operator's yaml files to add Docker inside them. Running Beam workers via Docker on the Flink nodes is not recommended (and probably not

Getting Beam(Python)-on-Flink-on-k8s to work

Hi folks, I'm still working with Pachama right now; we have a Kubernetes Engine cluster on GCP and want to run Beam Python batch pipelines with custom containers against it. Flink and Cloud Dataflow are the two options; Cloud Dataflow doesn't support custom containers for ba