Hello,
Thank you for preparing a new version. We should get rid of Java 8 as
soon as possible. There are several libraries that cannot be updated
because of Java 8. I tend to prefer to switch directly to JDK17. This is
supported by Spark 3.5, for example. Unfortunately, Hadoop 3 does not
work with JDK 17.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
Please keep in mind that Java 8 has a much longer support area than 11.
Most Linux distributions use openjdk and some people need to do the
backport process. RedHat as a contributor is dropping support for JDK 11
next month. https://endoflife.date/redhat-build-of-openjdk
Other companies like Microsoft seem to support OpenJDK 11 longer.
https://endoflife.date/microsoft-build-of-openjdk
I have seen the pull request with Docker examples. But I don't know if
we should focus on Zeppelin in Docker. The current implementation should
also be improved. The following start command doesn't look so good ...
https://github.com/apache/zeppelin/blob/940cf13935ec9ed989c7afeec98447c2e7eb5a7e/zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java#L194-L201
The development of our used docker-client library
`com.spotify:docker-client` was stopped a while ago.
https://github.com/spotify/docker-client
Spotify has probably switched from its own product Helios
(https://github.com/spotify/helios) to Kubernetes. I have been running
Zeppelin in Kubernetes for a long time. Getting started with Docker is
of course much easier.
The raft implementation library is also no longer being developed and we
should remove it. https://github.com/atomix/atomix-archive
This will also eliminate a lot of complexity. Maybe we can keep
interfaces so that another implementation is possible via plugin, for
example.
The EOL versions of Spark and Flink should be removed to speed up CI and
reduce complexity.
As written on GitHub, I like the switch to the new user interface. I
hope that development will be easier with a newer Angular stack.
Due to the many changes, I prefer a 0.12 version.
Best Regards
Philipp Dallig
Am 11.09.24 um 12:30 schrieb Adam Binford:
+1 for deprecating Java 8 and dropping Spark 3.2.
I also have a working version of "unofficial" Spark 4.0 support I can
submit a PR for. Unofficial because I don't want to mark 4.0.0 as
supported before it's released, but I got the Spark interpreter tests
to pass with 4.0.0-preview1.
One thing I noticed while trying to build a release with Spark 4.0
support is that the 0.12.0-SNAPSHOT build I made broke in our
deployment, because it looks like there was a change that forced
keytab/principal settings to only be loaded from env vars or system
props, and not the zeppelin-site.xml config. I ended up just
backporting it to a 0.11.2 build and Spark 4.0.0-preview1 seems to be
working fine.
Adam
On Wed, Sep 11, 2024 at 2:21 AM Cheng Pan <pan3...@gmail.com> wrote:
Thanks for bringing up this discussion, generally, I agree that we
should have a short release cycle.
For the version number, I prefer to 0.12 instead of 1.0, because
some interpreters relies on external systems were not fully tested
with Java 11, I would like to give it more time to collect user
feedbacks before enter 1.0.
+1 for marking Java 8 deprecated in the next release, meanwhile,
if someone is interested, we should also consider adding support
for Java 17, to prepare support for Spark 4.0.
I would also like to propose dropping support for Spark 3.2, and
making Spark 3.5 as the default version.
Currently, Zeppelin supports Spark 3.2 to 3.5.
Spark 3.2 released at Oct 2021, was EOL at Apr 2023.
Spark 3.3 released at Jun 2022, was EOL at Dec 2023.
Spark 3.4 released at Apr 2023, will be EOL at Oct 2024.
Spark 3.5 released at Aug 2024, it’s a LTS version.
For users who want to use lower Spark versions, Livy and
Kyuubi(for now, only supports Spark SQL via the JDBC interpreter)
are alternatives.
Thanks,
Cheng Pan
> On Sep 11, 2024, at 10:24, Jongyoul Lee <jongy...@gmail.com> wrote:
>
> Hello,
>
> I would like to discuss a new release with the community.
>
> Here is what we are working on:
> - Docker enhancement
> - Java 8 deprecation
> - New UI
> - Raft deprecation
>
> If you have any questions or ideas for the next release please feel
> free to leave your comments.
>
> Focusing on contributing and reviewing relevant tasks, I hope we do
> the next release by the end of this month.
>
> --
> Best regards,
> Jongyoul Lee
--
Adam Binford
