This is super helpful, thank you! I was looking for these kinds of switches in the other build configs, I agree that a generalization of this would be helpful going forward (thinking about other libraries that might require higher Java versions in the future). Thank you for taking the time to test my PR, I will incorporate your Java 17/21 check into it and see about the stuck integration test.
On Wed, Apr 30, 2025 at 6:08 PM Yi Hu <ya...@google.com> wrote: > Yeah those configurations are needed to accommodate errorprone. I did some > test on https://github.com/apache/beam/pull/34788 based on your PR. I get > Debezium build on CI and unit test passing (except there is an integration > test stuck). > > > On Wed, Apr 30, 2025 at 11:46 AM Tobi Kaymak <kay...@google.com> wrote: > >> Thank you! Yes, that is my understanding too. >> I tried to force Java 17, in Debezium IOs build.gradle, but I failed so >> far. >> I even tried to completely clear the arguments and manually construct >> them for the gradle command line [0], however then this clashed >> with ErrorProne. >> I will give it another shot in the coming days. >> >> [0] >> https://github.com/apache/beam/pull/34763/commits/a779e201941ba10672e8225229bc59e0afd01e21 >> >> On Wed, Apr 30, 2025 at 5:18 PM Yi Hu <ya...@google.com> wrote: >> >>> The purpose of the fallback [0] is to make other Beam components build >>> on Java11, while only `:sdks:java:io:debezium` should be configured to >>> build on Java17, in its build.gradle file. >>> >>> On Wed, Apr 30, 2025 at 11:14 AM XQ Hu via dev <dev@beam.apache.org> >>> wrote: >>> >>>> I think we can remove 11. Looks like we always do this: JAVA_HOME: >>>> /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/11.0.27-6/x64 >>>> >>>> However, we probably need to change that direct github workflow to >>>> remove :sdks:java:container:java8 and :sdks:java:container:java11 and >>>> add :sdks:java:container:java17. >>>> >>>> On Wed, Apr 30, 2025 at 10:23 AM Tobi Kaymak <kay...@google.com> wrote: >>>> >>>>> I do have a silly question, I am struggling to find the reason for the >>>>> fallback need of Java 11 in the beam_PreCommit_Java_Debezium_IO_Direct.yml >>>>> [0] >>>>> I was trying to not interfere with the build system outside of the >>>>> module by doing some tricks within the build.gradle of Debzium IO in the >>>>> second PR. [1] >>>>> However, it seems that the CI always picks up Java 11 and therefore >>>>> fails the build. It builds completely fine with Java 17 on my machine >>>>> though. >>>>> Is there a way to force Java 17 for the module by, for example, >>>>> removing the 11 fallback from the PreCommit Workflow or does this have >>>>> known side-effects? >>>>> >>>>> Thank you! >>>>> >>>>> [0] >>>>> https://github.com/apache/beam/pull/34751/files#diff-de752bcab62398ee28adb75a8fdb1933d8d855f2ae46417ec1d47180174a1800R91 >>>>> [1] https://github.com/apache/beam/pull/34763 >>>>> >>>>> On Mon, Apr 28, 2025 at 5:04 PM Tobi Kaymak <kay...@google.com> wrote: >>>>> >>>>>> I did the second PR and removed the enforcement of proto3 in the >>>>>> build.gradle ( >>>>>> https://github.com/apache/beam/pull/34763/files#diff-1572bcc322ede7d3a757381b84791a29b691df4d8947f6ad39ce4597653787edL95) >>>>>> - >>>>>> All tests are green on my local machine, but they will of course fail >>>>>> on the CI since the first PR has not been merged yet. >>>>>> >>>>>> I would be very happy about a review by Danny McCormick, who did work >>>>>> on this a few months ago in his PR and added the enforcement ( >>>>>> https://github.com/apache/beam/pull/33526). >>>>>> >>>>>> >>>>>> On Sun, Apr 27, 2025 at 5:00 PM Tobi Kaymak <kay...@google.com> >>>>>> wrote: >>>>>> >>>>>>> I did create the first PR, after building the IO on my local machine >>>>>>> with gradle and Java 17. This should not have an effect other than >>>>>>> changing >>>>>>> the build system. Feedback is very welcome: >>>>>>> https://github.com/apache/beam/pull/34751 >>>>>>> >>>>>>> On Sat, Apr 26, 2025 at 12:49 AM Tobi Kaymak <kay...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thank you both! >>>>>>>> I have checked and couldn't find a matching issue, so I created >>>>>>>> this one: https://github.com/apache/beam/issues/34747 >>>>>>>> >>>>>>>> On Sat, Apr 26, 2025 at 12:32 AM XQ Hu via dev <dev@beam.apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Can we create the GitHub issue to track this work if we haven't? >>>>>>>>> >>>>>>>>> On Fri, Apr 25, 2025, 4:39 PM Yi Hu via dev <dev@beam.apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Tobi, >>>>>>>>>> >>>>>>>>>> Thanks for your interest and willing to take on the task. First >>>>>>>>>> of all, could you please create a GitHub Issue to track the effort? >>>>>>>>>> >>>>>>>>>> I would like to share some comments if find useful >>>>>>>>>> >>>>>>>>>> - We already have infrastructure for the goal of "PR 1". We use >>>>>>>>>> BeamPluginModulePlugin [1] to initialize java modules and there is a >>>>>>>>>> setJavaVerOptions method. Here is a module could be an example how >>>>>>>>>> compiling with different Java version works in Beam: [2]. >>>>>>>>>> >>>>>>>>>> - We'll need to handle Debezium with newer Java version as well >>>>>>>>>> (That's the reason test currently failing on #34733). We'll need to >>>>>>>>>> modify >>>>>>>>>> test workflow file [3]. However due to GitHub limitation changes to >>>>>>>>>> .github/workflows won't be effective on unsubmitted PRs. You'd need >>>>>>>>>> to test >>>>>>>>>> your change locally. >>>>>>>>>> >>>>>>>>>> - Given that current Debezium (1.x) is already old I'm +1 with >>>>>>>>>> the change. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy >>>>>>>>>> >>>>>>>>>> [2] >>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/container/agent/build.gradle >>>>>>>>>> >>>>>>>>>> [3] >>>>>>>>>> https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Yi >>>>>>>>>> >>>>>>>>>> On Fri, Apr 25, 2025 at 3:06 PM Tobi Kaymak via dev < >>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Beam Devs, >>>>>>>>>>> >>>>>>>>>>> I'm interested in helping to upgrade the Debezium IO connector ( >>>>>>>>>>> sdks:java:io:debezium) to Debezium version 3.1.1.Final. This >>>>>>>>>>> version of Debezium requires its dependencies (like >>>>>>>>>>> debezium-core) to be compiled against Java 17 >>>>>>>>>>> <https://debezium.io/releases/3.0/release-notes#breaking_changes_14> >>>>>>>>>>> bytecode. >>>>>>>>>>> >>>>>>>>>>> I took the work from Danny McCormick who started doing something >>>>>>>>>>> similar for Debezium 2.7.4 and created a WIP branch >>>>>>>>>>> <https://github.com/apache/beam/pull/34733> to get my feet wet >>>>>>>>>>> with Java and Gradle's build system again. >>>>>>>>>>> >>>>>>>>>>> To manage this upgrade gracefully and make the review process >>>>>>>>>>> smoother, I was thinking to split the work into a two-step PR: >>>>>>>>>>> >>>>>>>>>>> 1. >>>>>>>>>>> >>>>>>>>>>> *PR 1 (Build Infrastructure):* A smaller, focused PR to >>>>>>>>>>> enable Java 17 compilation specifically for the >>>>>>>>>>> sdks:java:io:debezium module. This would likely involve: >>>>>>>>>>> - Modifying the sdks:java:io:debezium/build.gradle to allow >>>>>>>>>>> forking its compileJava (and compileTestJava) tasks using >>>>>>>>>>> a JDK 17 (e.g., via a java17Home Gradle property derived >>>>>>>>>>> from a CI environment variable like JAVA_HOME_17_X64). >>>>>>>>>>> - Updating the relevant PreCommit workflow ( >>>>>>>>>>> beam_PreCommit_Java_Debezium_IO_Direct.yml) >>>>>>>>>>> >>>>>>>>>>> <https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml#L87> >>>>>>>>>>> to ensure a JDK 17 is available and its path is passed to >>>>>>>>>>> Gradle for this >>>>>>>>>>> module. >>>>>>>>>>> - The goal of this first PR would be to get the module to >>>>>>>>>>> compile successfully with Java 17 against the existing >>>>>>>>>>> (older) Debezium >>>>>>>>>>> version. >>>>>>>>>>> 2. >>>>>>>>>>> >>>>>>>>>>> *PR 2 (Debezium 3.1.1 Upgrade & API Adaptation):* A >>>>>>>>>>> subsequent PR that would: >>>>>>>>>>> - Actually update the Debezium dependencies to 3.1.1.Final. >>>>>>>>>>> - Incorporate the necessary code changes within >>>>>>>>>>> sdks:java:io:debezium to adapt to any API changes in >>>>>>>>>>> Debezium 3.1.1. >>>>>>>>>>> - Address any new functionality or considerations with >>>>>>>>>>> Debezium 3.1.1 (which I understand uses now libprotoc-dev >>>>>>>>>>> 1.5, >>>>>>>>>>> >>>>>>>>>>> <https://github.com/debezium/container-images/pull/427/files#diff-bf3b0b8b023cf3cd20ca9d862bb8301523cdb9c934dec1ce96cd9640a613ae7e> >>>>>>>>>>> - >>>>>>>>>>> according to its release notes >>>>>>>>>>> >>>>>>>>>>> <https://debezium.io/releases/3.1/release-notes#other_changes_5> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>>> My reasoning for this split is to isolate the build system >>>>>>>>>>> changes from the actual library upgrade and code adaptation, >>>>>>>>>>> allowing for >>>>>>>>>>> smaller code reviews. >>>>>>>>>>> >>>>>>>>>>> I'd appreciate any feedback or suggestions on the best way to >>>>>>>>>>> upgrade Debezium IO. >>>>>>>>>>> >>>>>>>>>>> Best >>>>>>>>>>> Tobi >>>>>>>>>>> >>>>>>>>>>
