Ekaterina, thank you very much for sharing this! I admit, it’s much more involved than I expected to be.
The —add-opens and —add-exports flags seem suitable for development and perhaps experimental support, but we’ll probably want to make changes to remove as many as we can before considering JDK17 suitable for general use. Updating dependencies will be good for general hygiene, but it would also be important for us to scan changelogs for the intervening versions of each we’re upgrading to flag any unexpected behavior changes that may impact Cassandra or which the community might want to know about. Thank you for working to get Cassandra launching on JDK17. Excited for what’s ahead. — Scott > On Mar 23, 2022, at 7:58 AM, Ekaterina Dimitrova <e.dimitr...@gmail.com> > wrote: > > > Hi everyone, > > Looking into our way to Java 17, I wanted to share with the community > findings/thoughts and align on course of action. > > We already deprecated scripted UDFs so we can remove them when the time to > switch from Java8&11 to Java 11&17 comes. I removed the ant script tasks and > created custom ant tasks to workaround the need of Nashorn. Ant is also > upgraded now on trunk. > > With this Cassandra trunk compiles with warnings about the Security manager > being deprecated and other security deprecation warnings(I mention it for > awareness here). > I pushed to my personal docker hub account a version of our testing image > that has Java 17 installed and I worked on the build file, shell scripts and > config to push testing with Java 17 to CircleCI. > > To just start Cassandra out of the box on Java 17 we also need as a least > minimum to add the following, further to what we already open in Java 11: > --add-opens java.base/sun.nio.ch=ALL-UNNAMED > --add-opens java.base/java.io=ALL-UNNAMED > --add-opens java.base/java.util.concurrent=ALL-UNNAMED > --add-opens java.base/java.util=ALL-UNNAMED > --add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED > --add-opens java.base/java.nio=ALL-UNNAMED > --add-opens java.base/java.lang=ALL-UNNAMED > > --add-exports java.base/sun.nio.ch=ALL-UNNAMED > --add-exports java.base/java.io=ALL-UNNAMED > --add-exports java.base/java.util.concurrent=ALL-UNNAMED > --add-exports java.base/java.util=ALL-UNNAMED > --add-exports java.base/java.util.concurrent.atomic=ALL-UNNAMED > --add-exports java.base/java.nio=ALL-UNNAMED > --add-exports java.base/java.lang=ALL-UNNAMED > > This is a quick run of jdeps -jdkinternals --multi-release 17 > apache-cassandra-4.1-SNAPSHOT.jar.: > apache-cassandra-4.1-SNAPSHOT.jar -> java.rmi > apache-cassandra-4.1-SNAPSHOT.jar -> jdk.unsupported > org.apache.cassandra.io.util.Memory -> sun.misc.Unsafe > JDK internal API (jdk.unsupported) > org.apache.cassandra.utils.FastByteOperations$UnsafeOperations -> > sun.misc.Unsafe JDK internal API > (jdk.unsupported) > org.apache.cassandra.utils.FastByteOperations$UnsafeOperations$1 -> > sun.misc.Unsafe JDK internal API > (jdk.unsupported) > org.apache.cassandra.utils.JMXServerUtils$JmxRegistry -> > sun.rmi.registry.RegistryImpl JDK internal API (java.rmi) > org.apache.cassandra.utils.memory.MemoryUtil -> sun.misc.Unsafe > JDK internal API (jdk.unsupported) > > Warning: JDK internal APIs are unsupported and private to JDK implementation > that are > subject to be removed or changed incompatibly and could break your > application. > Please modify your code to eliminate dependence on any JDK internal APIs. > For the most recent update on JDK internal API replacements, please check: > https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool > > Also a quick workaround I applied for test purposes in order to be able to > run the in-jvm tests can be seen here[4], and I assume something like that is > also needed for the simulator and other places which I went ahead and changed > in order just to unblock the preliminary testing so we can get the full > picture. To be revised later. > This was the issue we were hitting: > java.lang.RuntimeException: java.lang.NoSuchFieldException: modifiers > at > org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:691) > at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81) > at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47) > at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > Caused by: java.lang.NoSuchFieldException: modifiers > at java.base/java.lang.Class.getDeclaredField(Class.java:2610) > at org.apache.cassandra.net.Verb.unsafeSetSerializer(Verb.java:366) > at > org.apache.cassandra.distributed.impl.Instance.lambda$startup$11(Instance.java:612) > > To fire testing I used some config suggested by Jonathan Shook which he used > for testing with Java 16 some time ago[1]. We will need to tune it, those > were used to do preliminary work and initial investigations only. > > One important topic I want to bring to the community's attention is our > dependency management. [2] is a discussion we had a long time ago when I > joined the project. So is it correct to say for Java 17 we don't update > anything if there is no immediate need (and this considers only trunk of > course)? In my branch I updated bytebuddy, asm, chronicle queues, byteman, > mockito, jacoco, ecj, sjk so far. > > Based on feedback from CASSANDRA-17392 I tried to update netty and run our > tests with Java 17 and I ran into java.lang.ClassCastException: class > org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class > sun.nio.ch.DirectBuffer . I didn't have the chance to investigate it but it > is on our table so good to be mentioned. > > Even with the update of Chronicle queues[3] we still have to add for FQLtool > and AuditLogging: > --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ > --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \ > --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \ > --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED \ > --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \ > --add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED \ > --add-opens=java.base/java.lang=ALL-UNNAMED \ > --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \ > --add-opens=java.base/java.io=ALL-UNNAMED \ > --add-opens=java.base/java.util=ALL-UNNAMED\ > After that, from what I saw there is only one type of failure related to > those tools to be investigated that cannot be fixed directly with some other > internals opening. > > I also updated Jolokia in the DTest repo which seemed to solve any Jolokia > related problems, at least our tests are not failing due to Jolokia related > problems from what I saw so far but officially I see Jolokia builds running > on JDK 8, 9, and 11. I didn't find any info around Java 17 and Jolokia so far. > > These are a few of the things we hit now, so I wanted to mention in order to > give you an idea of the direction the things are going. > > So I am interested to hear any comments, particularly on our approach to > dependencies updates as a project. Also, how much are we willing to extend > the usage of --add-opens and --add-exports at this moment? The Java community > recommendation is this to be our last option. We have some dependencies which > still do not officially support Java 17 or which are not even fully tested > with Java 17. Some of them will work out of the box, some will need more > internals to be opened maybe or some other actions to be taken, I still don't > know the answers to all failures I see and sometimes I fix one but another > one appears. There are moving parts and I also didn't go ahead and update all > dependencies based on the previous discussions. (I also saw that when Java 11 > was introduced we didn't update all dependencies?) All of this is only to be > able to bring our CI to some stable condition on Java 17 as a starting point > and get an idea of what is needed. > > Also, I guess soon we can join the OpenJDK Quality Outreach program[5] > suggested before as we get more details that can be already discussed. I can > go ahead and do it if no one is against it, the contact can be just our > dev-mailing list I guess. > > [1]https://jaxenter.com/apache-cassandra-java-174575.html > [2]https://lists.apache.org/thread/l1ch2ffcnv2m5l9tp9mpvgzwlvx667sd > [3]https://chronicle.software/chronicle-support-java-17/ > [4]https://github.com/apache/cassandra/commit/4daf0b149bc1524dfb2212ab958f4d45feb79577 > [5]https://wiki.openjdk.java.net/display/quality/Quality+Outreach > > Ekaterina Dimitrova
