I'm still working on this and have created a WIP pull request to demonstrate the list of changes: https://github.com/apache/arrow/pull/43050
Laurent On Mon, May 6, 2024 at 9:40 AM Laurent Goujon <laur...@dremio.com> wrote: > I wanted to share some status updates: I do have a list of changes to > build Arrow with a recent version of Java using the `--release` plugin > option of javac. > The full change is available at: https://github.com/laurentgo/arrow/pull/3 > > Here are roughly how the changes are divided: > * Change access to MemoryUtil.UNSAFE to private, and replace direct > invocation of sun.misc.Unsafe methods with MethodHandle instances. The > reason for this change is that sun.misc.Unsafe class is not available when > using `--release` option (see https://bugs.openjdk.org/browse/JDK-8206937) > * Update some maven plugin dependencies to latest versions like > maven-compiler-plugin > * Add a new profile to allow for cross-version testing. The profile is > enabled by using `-Darrow.test.jdk-version=<jdk version to use>` where > version can be 1.8, 11, 17, ... > > Next steps: integrate the change with Arrow CI/CD environment (which is > why the change was not posted as a pull request against apache/arrow > repository yet). The main issue is that linux builds do not happen directly > within the GH runner (like macos or windows build) but inside a container. > Unfortunately there's only one JDK version available within the container, > so my next step would need to find a way to extend the image or create a > new one. If people have other options in mind, I'm also all ear. > > Laurent > > On Wed, Apr 10, 2024 at 9:07 AM Laurent Goujon <laur...@dremio.com> wrote: > >> I can give it a try for sure >> >> On Fri, Apr 5, 2024 at 10:26 AM Dane Pitkin <d...@voltrondata.com.invalid> >> wrote: >> >>> I think we can revisit the discussion soon for dropping Java 8 >>> altogether, >>> since Spark will release 4.0 in ~June supporting Java 17+ at runtime. >>> >>> I'm curious how big of an effort it would be to get your proposal >>> implemented. Would you be willing to draft a PR so we can see what types >>> of >>> changes are necessary? >>> >>> On Wed, Apr 3, 2024 at 8:05 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>> > Yes, correct for language features. My point was more that we can >>> > decide on a major Arrow version upgrading the target language version. >>> > That's what I meant by "consensus". >>> > >>> > Regards >>> > JB >>> > >>> > On Tue, Apr 2, 2024 at 5:55 PM Laurent Goujon >>> > <laur...@dremio.com.invalid> wrote: >>> > > >>> > > At code level we need to separate language features from library >>> > features? >>> > > It should be possible to leverage memory API for example through >>> > reflection >>> > > and/or multi-release jar files, but record is a language feature and >>> it >>> > > would not possible to use it without targeting java 17 at the source >>> > level. >>> > > >>> > > Laurent >>> > > >>> > > On Tue, Apr 2, 2024 at 1:40 AM Jean-Baptiste Onofré <j...@nanthrax.net >>> > >>> > wrote: >>> > > >>> > > > Hi Laurent >>> > > > >>> > > > It makes sense to me. I started this "move" (on the plugin side of >>> the >>> > > > thing) as part of the reproducible build effort. >>> > > > >>> > > > At code level, I think it would be great to leverage some features >>> > > > from Java 17+ (I'm thinking about record, memory API, etc). >>> > > > I would be more than happy to help on this as soon as we have a >>> > consensus. >>> > > > >>> > > > Thanks, >>> > > > Regards >>> > > > JB >>> > > > >>> > > > On Mon, Apr 1, 2024 at 7:48 PM Laurent Goujon >>> > > > <laur...@dremio.com.invalid> wrote: >>> > > > > >>> > > > > Hello Arrow Java developers, >>> > > > > >>> > > > > I would wonder if the community would be okay to change the >>> minimum >>> > Java >>> > > > > version used by the build toolchain to at least Java 17 or 21 (or >>> > even >>> > > > 22). >>> > > > > This is different from changing the minimum Java version used at >>> > runtime >>> > > > > which would still be 8 (following the vote from last september). >>> > > > > >>> > > > > Concretely it would mean: >>> > > > > * Java 21 would be required to build Arrow Java >>> > > > > * But Arrow would still be compatible with Java 8 >>> > > > > * Unit tests should keep running with Java 8 and higher. >>> > > > > >>> > > > > Reasons for changing the toolchain would be: >>> > > > > - More and more tools and plugins now require at least Java 11, >>> > forcing >>> > > > the >>> > > > > project to keep using older/unsupported versions >>> > > > > - We hacked our way to support Java modules with >>> > > > > >>> > > > >>> > >>> https://github.com/apache/arrow/tree/main/java/maven/module-info-compiler-maven-plugin >>> > > > > - There are several new features which we could conditionally >>> > include in >>> > > > > the Arrow project like VarHandle (Java 9) and Foreign Function >>> and >>> > Memory >>> > > > > API (Java 22) to move away from Unsafe support which require >>> more and >>> > > > more >>> > > > > workarounds (Apache Lucene is a project which has managed to >>> > introduce >>> > > > > support for multiple Java incubator and final API while >>> maintaining >>> > > > > compatibility with previous Java versions). But doing it from >>> Java 8 >>> > > > > creates a higher barrier. >>> > > > > >>> > > > > Laurent >>> > > > >>> > >>> >>
