Hi, James, Sorry to reply late. I just came back from a trip and had a discussion with our internal team as well. So, there is one proposal other than creating a branch. Let me elaborate it below: a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be the hosting module for most of the jdk11 related changes. b) modify the build script s.t. samza-yarn3 will only compile and build with jdk11 and samza-yarn only compile and build with jdk8. Thus, we can have two builds: jdk8 build that builds with samza-yarn w/ YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0. We can manage to publish both jdk8 and jdk11 artifacts if needed. The benefit of this approach is that we can still maintain the trunk release while opening up the jdk11 support.
Let me know if that works for you and we can work together to get the code in. Best! -Yi On Tue, Sep 13, 2022 at 7:53 AM James DeMichele <james.demich...@redfin.com.invalid> wrote: > Hey Yi, I wanted to follow up here and figure what a path forward is here. > We need to move to Java 11, and Samza currently is our only blocking issue. > In order to move to Java 11, the Yarn Cluster would need to run on Java 11 > correct? If that's the case, then it would need to be 3.3+. I don't know > what it entails on your end to have a new Major version, but that seems > like a good option here right? Version 2 could be where we can move this > project forward to Java 11, while Version 1 can still remain, and would not > break people that can't/won't upgrade to Java 11. > > -Jamie > > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <james.demich...@redfin.com > > > wrote: > > > Yeah I mean if Samza works fine with the hadoop-yarn library running > > against a 3.3.x YARN cluster, then I don't mind keeping that library of > > 2.10.x in Samza's code. But it is still a moot point in terms of > upgrading > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be > able > > to run the Cluster with Java 11. > > > > @Yi, I think that moving to a new major version might be the solution > > here. That way Linkedin can still have a pathway of upgrading code for > the > > old legacy 1.x version of Samza. While a new major version of 2.x of > Samza > > could then make it a requirement that it runs with a YARN cluster of > 3.3.x > > if you want to use Java 11. > > > > The only issue there is that you'll probably need to backport changes > > between the 2 versions. But in all honestly, this project does not look > > extremely active with commits so it might not be that big of a problem. > > > > -Jamie > > > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mmcfarl...@cavulus.com > > > > wrote: > > > >> Hi all, > >> > >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4; > >> afaict, in light testing, Samza seems to work fine using the 2.10.x > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie > pointed > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 ( > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit > >> tests > >> in Samza that verify compatibility against a YARN cluster? If so, that > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility. > >> > >> Just throwing my 2 cents out there, > >> Malcolm McFarland > >> Cavulus > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele > >> <james.demich...@redfin.com.invalid> wrote: > >> > >> > Hey Yi, > >> > > >> > Thanks for getting back to me. I have not tried the older yarn cluster > >> > version yet in the Samza app running against 3.3.4 but I am wary it > >> would > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you > would > >> > have to update yarn even if the Java library here wasn't updated. > >> > > >> > Could we move this version I'm proposing to a 2.x version of Samza? So > >> > people that wanted to move forward with yarn upgrade and Samza and > Java > >> 11 > >> > (like us) could do so? Then 1.x could only be java 8 compatible and > 2.x > >> > could be java 11. > >> > > >> > Jamie > >> > > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com> wrote: > >> > > >> > > Hey, James, > >> > > > >> > > Thanks for the ping. @prateek, can we have someone to review this > >> change? > >> > > > >> > > One question: have you tested the change w/ the older YARN cluster > >> > version > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to > >> 3.3.4 > >> > as > >> > > well, that may be a breaking change to existing Samza users (i.e. > >> > LinkedIn > >> > > is still running a YARN cluster with version 2.10.1). > >> > > > >> > > Best and apologize for the delay. > >> > > > >> > > -Yi > >> > > > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele > >> > > <james.demich...@redfin.com.invalid> wrote: > >> > > > >> > > > Hey y'all. I just am not sure how to get some traction on these > >> Java 11 > >> > > > PRs. > >> > > > > >> > > > https://github.com/apache/samza/pull/1628 > >> > > > https://github.com/apache/samza-hello-samza/pull/87 > >> > > > > >> > > > Would someone that is a maintainer for Samza just let us know that > >> > y'all > >> > > > are looking at them? I can stop pestering you :) > >> > > > > >> > > > I ran all tests in both PRs, all pass. I also confirmed that using > >> my > >> > > Samza > >> > > > PR in the Hello World app all works with Java 11. > >> > > > > >> > > > Thanks! > >> > > > > >> > > > -Jamie > >> > > > > >> > > > >> > > >> > > >
