Adam, good to know that you are working on Spark 4 support, and Spark 4.0 preview2 is likely coming soon. It would be great if you can submit a draft PR even Spark 4.0 is not GA.
> it looks like there was a change that forced keytab/principal settings to > only be loaded from env vars or system props, and not the zeppelin-site.xml this sounds like a regression that should be fixed, I’m not sure what’s the issue exactly being, if you are willing to debug, this project[1] can leverage docker compose to simplify setup a kerberized Hadoop environment. [1] https://github.com/awesome-kyuubi/hadoop-testing Thanks, Cheng Pan > On Sep 11, 2024, at 18:30, Adam Binford <adam...@gmail.com> wrote: > > +1 for deprecating Java 8 and dropping Spark 3.2. > > I also have a working version of "unofficial" Spark 4.0 support I can > submit a PR for. Unofficial because I don't want to mark 4.0.0 as supported > before it's released, but I got the Spark interpreter tests to pass with > 4.0.0-preview1. > > One thing I noticed while trying to build a release with Spark 4.0 support > is that the 0.12.0-SNAPSHOT build I made broke in our deployment, because > it looks like there was a change that forced keytab/principal settings to > only be loaded from env vars or system props, and not the zeppelin-site.xml > config. I ended up just backporting it to a 0.11.2 build and Spark > 4.0.0-preview1 seems to be working fine. > > Adam > > On Wed, Sep 11, 2024 at 2:21 AM Cheng Pan <pan3...@gmail.com> wrote: > >> Thanks for bringing up this discussion, generally, I agree that we should >> have a short release cycle. >> >> For the version number, I prefer to 0.12 instead of 1.0, because some >> interpreters relies on external systems were not fully tested with Java 11, >> I would like to give it more time to collect user feedbacks before enter >> 1.0. >> >> +1 for marking Java 8 deprecated in the next release, meanwhile, if >> someone is interested, we should also consider adding support for Java 17, >> to prepare support for Spark 4.0. >> >> I would also like to propose dropping support for Spark 3.2, and making >> Spark 3.5 as the default version. >> >> Currently, Zeppelin supports Spark 3.2 to 3.5. >> >> Spark 3.2 released at Oct 2021, was EOL at Apr 2023. >> Spark 3.3 released at Jun 2022, was EOL at Dec 2023. >> Spark 3.4 released at Apr 2023, will be EOL at Oct 2024. >> Spark 3.5 released at Aug 2024, it’s a LTS version. >> >> For users who want to use lower Spark versions, Livy and Kyuubi(for now, >> only supports Spark SQL via the JDBC interpreter) are alternatives. >> >> Thanks, >> Cheng Pan >> >> >> >>> On Sep 11, 2024, at 10:24, Jongyoul Lee <jongy...@gmail.com> wrote: >>> >>> Hello, >>> >>> I would like to discuss a new release with the community. >>> >>> Here is what we are working on: >>> - Docker enhancement >>> - Java 8 deprecation >>> - New UI >>> - Raft deprecation >>> >>> If you have any questions or ideas for the next release please feel >>> free to leave your comments. >>> >>> Focusing on contributing and reviewing relevant tasks, I hope we do >>> the next release by the end of this month. >>> >>> -- >>> Best regards, >>> Jongyoul Lee >> >> > > -- > Adam Binford
