Re: Dataproc+Flink+Beam Portable Runner Tutorial

2021-07-13 Thread Joey Tran
Where are those? Yeah I saw Dataflow but unfortunately we have a use case that requires custom image and higher scale than the dataflow quota allows On Tue, Jul 13, 2021 at 2:00 PM Kyle Weaver wrote: > You can check the task manager logs as well to see if there is any > additional information.

Re: Dataproc+Flink+Beam Portable Runner Tutorial

2021-07-12 Thread Joey Tran
Yeah I saw the 1.9 and tried it initially but the cluster flink version is actually updated and in the 2.0 image flink==1.12 (I confirmed by looking at the flink dashboard from the YARN ResourceManager web interface). I checked out beam==2.30.0 and built :runners:flink:1.12:job-server:shadowJar an

Re: Dataproc+Flink+Beam Portable Runner Tutorial

2021-07-09 Thread Joey Tran
Hi kyle, I noticed the one you linked didnt work when i copied and pasted it so chose the only one in the directory and that’s how when i got the error message Thanks! Joey On Fri, Jul 9, 2021 at 4:50 PM Kyle Weaver wrote: > Hi Joey, my mistake, I picked the wrong jar. The correct jar should b

Re: Dataproc+Flink+Beam Portable Runner Tutorial

2021-07-09 Thread Joey Tran
Hi all, Thank you very much for the responses! I feel a bit better about using dataproc since it's not in beta like flink on GKE. I rebuilt the flinkrunner as you specified but I still get an error. I've attached the stdout from trying to run with the patched flink runner. Here's instructions t

Dataproc+Flink+Beam Portable Runner Tutorial

2021-07-09 Thread Joey Tran
Hello! I'm trying to just demo Beam/Flink and I tried following the instructions with Google's Dataproc but I get a bunch of errors ranging from jackson dependency issues to some issue about "No container id". Does anyone know if these dataproc instructions[1] are complete? I ran through it prett