I believe our usage is high enough on Java 8 that we wouldn't be close to ceasing support for it.
I actually don't really know much about the possibility of distributing multi-version artifacts for Java but that would probably be better than what we do today. Today we just distribute Java 8 artifacts that will then work with other versions. Kenn On Fri, Jul 21, 2023 at 11:01 PM Damon Douglas <damondoug...@apache.org> wrote: > Thank you everyone for responding. > > Replying to Kenn: > > 1. This explanation helps me understand more. > 2. I realize the drive for the breaking change PR in question is to > support Java 17. Would a potential path be to target a more gentle ramp > from 8, to 11, and up with a communicated and anticipated date of ceasing > support for a particular version? > > On Fri, Jul 21, 2023 at 9:51 AM Reuven Lax via dev <dev@beam.apache.org> > wrote: > >> Curious why these failing tests didn't block submission. >> >> For now rollback seems to be the simplest option. However is there a path >> forward on Java 11, or is our model irretrievably broken on Java 11? >> >> On Fri, Jul 21, 2023 at 8:57 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> This is a tricky situation that I don't know how to resolve best. Here >>> are some pieces of information I know: >>> >>> 1. The reason we put certain generated classes in the same package is >>> because of Java's package-private access restriction. If they are in >>> another package the generated wrapper won't be able to invoke the needed >>> functions. I know this applies to a generated DoFnInvoker. I don't know if >>> it applies here. >>> >>> 2. The current status for Beam is that Beam itself is only >>> expected/required to be able to build with Java 8 and/or produce Java 8 >>> compatible bytecode, but users should be able to use it with their own Java >>> 11 or Java 17 code. This makes the testing scenario a bit tricky. We do >>> have tests that model this scenario but they did not catch this I guess. >>> >>> On Mon, Jul 17, 2023 at 1:19 AM Damon Douglas <damondoug...@apache.org> >>> wrote: >>> >>>> Good day, everyone, >>>> >>>> For clarity, I organize the following into situation, background, >>>> assessment, and proposal. >>>> >>>> Best, >>>> >>>> Damon >>>> >>>> ----------------------------------------- >>>> >>>> Situation >>>> >>>> Issue #26981 reports an IllegalArgumentException associated with the >>>> ByteBuddy dependency throwing the message "<some class> must be defined in >>>> the same package as <some other class>"[1]. I personally discovered this >>>> error blocking my own Schema-related tests. >>>> >>>> Background >>>> >>>> *1. PR #25578 introduced the error* >>>> >>>> As Issue #26981 reports[1], the error seems to be introduced with 2.48. >>>> Comparing v2.47.0 and v2.48.0[2] reveals that PR #25578 may have introduced >>>> this breaking change[3]. Said PR replaced ByteBuddy's >>>> ClassLoadingStrategy.Default.INJECTION[4] with getClassLoadingStrategy[5]. >>>> >>>> *2. Reverting PR #25578 resolves the error* >>>> >>>> To test this hypothesis, I cloned 41e6628 and ran: >>>> >>>> ./gradlew :sdks:java:core:check >>>> >>>> revealing several failing tests (see *Failing :sdks:java:core:check at >>>> 41e6628* below), some of which contained the >>>> familiar IllegalArgumentException "<some class> must be defined in the same >>>> package as <some other class>" message. >>>> >>>> After reverting changes found in #25578, the failing tests and the >>>> IllegalArgumentException were resolved. >>>> >>>> *3. Code related to PR #25578 has a back and forth history* >>>> >>>> There seems to be a back and forth removal and replacement history[6] >>>> between ByteBuddy's ClassLoadingStrategy.Default.INJECTION >>>> and getClassLoadingStrategy most recently PR #25578. Said PR's motivation >>>> is to prepare Beam for Java 17 compatibility, which explains the >>>> re-introduction of the breaking changes. >>>> >>>> *4. PR #25578 GitHub Actions checks pass* >>>> >>>> Examining the GitHub actions run reveals that PR #25578 checks >>>> passed[7]. However, examining the setup[8] more closely reveals that Java >>>> tests are executed using Java Version 8. The same is true in the >>>> latest 41e6628 commit[9]. To test whether the version of Java drives Issue >>>> #26981's error, I submitted a draft PR[10] with the version of Java set to >>>> 11 and found that the same errors resulted[11] as I found on my machine >>>> using the same Java version. >>>> >>>> Assessment >>>> >>>> My main impression is that: >>>> >>>> 1. checks did not reveal PR #25578's breaking changes[7] because >>>> the environment[8] used Java 8 instead of 11 >>>> 2. the back and forth removal and addition of PR #25578's changes >>>> does not solve current and future Java version compatibilities >>>> >>>> Proposal >>>> >>>> May we consider: >>>> >>>> 1. If not already planned, set >>>> .github/actions/setup-self-hosted-action/action.yml's Java version[12] >>>> to >>>> 11. >>>> 2. arriving at a consensus regarding PR #25578's breaking changes >>>> and what we need to do today and in the future; I don't have anything >>>> practical to propose or recommend >>>> >>>> References >>>> >>>> 1. https://github.com/apache/beam/issues/26981 >>>> 2. https://github.com/apache/beam/compare/v2.47.0...v2.48.0 >>>> 3. https://github.com/apache/beam/pull/25578 >>>> 4. >>>> >>>> https://javadoc.io/static/net.bytebuddy/byte-buddy/1.12.23/net/bytebuddy/dynamic/loading/ClassLoadingStrategy.Default.html#INJECTION >>>> 5. >>>> >>>> https://github.com/apache/beam/blob/68e19a596a5d0136ba4592be01888f487463c2f3/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ByteBuddyUtils.java#L32 >>>> 6. >>>> >>>> https://github.com/apache/beam/commits/68e19a596a5d0136ba4592be01888f487463c2f3/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ByteBuddyUtils.java >>>> 7. >>>> https://github.com/apache/beam/actions/runs/4759753495/jobs/8459342173 >>>> 8. >>>> >>>> https://github.com/apache/beam/actions/runs/4759753495/jobs/8459342173#step:3:28 >>>> 9. >>>> >>>> https://github.com/apache/beam/actions/runs/5558349809/jobs/10153336615#step:3:16 >>>> 10. >>>> >>>> https://github.com/apache/beam/actions/runs/5571881403/jobs/10177348958?pr=27515#step:3:16 >>>> 11. >>>> >>>> https://github.com/apache/beam/actions/runs/5571881403/jobs/10177348958?pr=27515#step:5:123 >>>> 12. >>>> >>>> https://github.com/apache/beam/blob/561870be80cc8696a926d7322b0f47ddeddcc222/.github/actions/setup-self-hosted-action/action.yml#L56 >>>> >>>> >>>> Failing :sdks:java:core:check at 41e6628 >>>> >>>> *Ran locally on*: >>>> >>>> openjdk 11.0.14.1 2022-02-08 >>>> OpenJDK Runtime Environment JBR-11.0.14.1.1-2043.11-jcef (build >>>> 11.0.14.1+1-b2043.11) >>>> OpenJDK 64-Bit Server VM JBR-11.0.14.1.1-2043.11-jcef (build >>>> 11.0.14.1+1-b2043.11, mixed mode) >>>> >>>> org.apache.beam.sdk.schemas.AvroSchemaTest > testPojoRecordToRow FAILED >>>> java.lang.IllegalArgumentException at AvroSchemaTest.java:451 >>>> >>>> org.apache.beam.sdk.schemas.AvroSchemaTest > testSpecificRecordToRow >>>> FAILED >>>> java.lang.IllegalArgumentException at AvroSchemaTest.java:386 >>>> >>>> org.apache.beam.sdk.schemas.JavaBeanSchemaTest > testMapFieldSetters >>>> FAILED >>>> java.lang.IllegalArgumentException at JavaBeanSchemaTest.java:460 >>>> >>>> org.apache.beam.sdk.schemas.JavaBeanSchemaTest > >>>> testRecursiveArraySetters FAILED >>>> java.lang.IllegalArgumentException at JavaBeanSchemaTest.java:385 >>>> >>>> org.apache.beam.sdk.schemas.JavaBeanSchemaTest > testMapFieldGetters >>>> FAILED >>>> java.lang.IllegalArgumentException at JavaBeanSchemaTest.java:437 >>>> >>>> org.apache.beam.sdk.schemas.JavaBeanSchemaTest > >>>> testRecursiveArrayGetters FAILED >>>> java.lang.IllegalArgumentException at JavaBeanSchemaTest.java:369 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > testNestedArraysToRow >>>> FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:602 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > testEnumFieldToRow >>>> FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:658 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > >>>> testRecursiveArraySetters FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:411 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > testMapFieldGetters >>>> FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:463 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > >>>> testRecursiveArrayGetters FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:394 >>>> >>>> org.apache.beam.sdk.schemas.JavaFieldSchemaTest > >>>> testNestedArraysFromRow FAILED >>>> java.lang.IllegalArgumentException at JavaFieldSchemaTest.java:580 >>>> >>>> org.apache.beam.sdk.schemas.utils.JsonUtilsTest > >>>> testGetRowToJsonBytesFunction FAILED >>>> java.lang.IllegalArgumentException at JsonUtilsTest.java:92 >>>> >>>> org.apache.beam.sdk.schemas.utils.JsonUtilsTest > >>>> testGetJsonStringToRowFunction FAILED >>>> java.lang.IllegalArgumentException at JsonUtilsTest.java:92 >>>> >>>> org.apache.beam.sdk.schemas.utils.JsonUtilsTest > >>>> testGetRowToJsonStringsFunction FAILED >>>> java.lang.IllegalArgumentException at JsonUtilsTest.java:92 >>>> >>>> org.apache.beam.sdk.schemas.utils.JsonUtilsTest > >>>> testGetJsonBytesToRowFunction FAILED >>>> java.lang.IllegalArgumentException at JsonUtilsTest.java:92 >>>> >>>
