Fwd: Beam Error downgrading from Flink 1.10 to 1.8

2020-06-15 Thread Roger
Hello. I am currently trying to get a Beam application to work on Flink version 1.8 that currently works on version 1.10. I realized that AWS kinesis does not support flink version 1.10 so I must downgrade. With that said when I downgrade I get a stacktrace that I don't really understand. Any help

Re: Beam Error downgrading from Flink 1.10 to 1.8

2020-06-15 Thread Roger
Whelp. I think I might have just solved my own problem. It looks like when I downgraded, I was still using flink-runner version 1.10 in my pom.xml. I downgraded this to 1.8 and it seems to have worked! On Mon, Jun 15, 2020 at 12:41 PM Roger wrote: > Hello. > I am currently trying to get a Beam a

Re: Beam supports Flink RichAsyncFunction

2020-06-15 Thread Luke Cwik
The intent is that users shouldn't have to use async I/O since the idea is that the runner should increase the number of workers/threads being processed automatically so that you never need to special case this. Unfortunately Dataflow is the only one who does this today so you are forced to use som

Re: Building Dataflow Worker

2020-06-15 Thread Luke Cwik
I noticed that you are not using the gradle wrapper but your own installed version. Apache Beam 2.19 is using gradle 5.2.1, is the installed version compatible with that? Try ./gradlew :runners:google-cloud-dataflow-java:worker:shadowJar in a clean workspace. On Fri, Jun 12, 2020 at 4:30 PM Talat

Re: Building Dataflow Worker

2020-06-15 Thread Steve Niemitz
I think you want the "legacy-worker" target instead: ./gradlew -Ppublishing -PnoSigning :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar That's what I've always used at least. On Mon, Jun 15, 2020 at 4:57 PM Luke Cwik wrote: > I noticed that you are not using the gradle wrapp

Re: Building Dataflow Worker

2020-06-15 Thread Talat Uyarer
Thank you Luke and Steve for your help. Looks like there is an issue with Gradle Errorprone's plugin. I had to modify BeamModulePlugin.groovy [1], Without modification when i tried to build it I got this error[2]. Should I create a patch for it ? Last question. We are using streaming engine. Whic

Re: Building Dataflow Worker

2020-06-15 Thread Kyle Weaver
> Looks like there is an issue with Gradle Errorprone's plugin. I saw the same error earlier today: https://jira.apache.org/jira/browse/BEAM-10263 I haven't done a full investigation yet, but it seems this broke the build for several (?) past Beam source releases. I'm not sure what Beam's policy