Re: [DISCUSS] Spark 3.1 support?

> without requiring authors to cherry-pick all applicable changes, like we agreed initially. Not trying to change what agreed before. Just for my understanding. Let's say the latest Spark version is 3.3. Today, we don't require any backport to 3.2 and 3.1, correct? On Fri, Apr 21, 2023 at 5:19 P

Re: [DISCUSS] Spark 3.1 support?

I still agree with the idea that people interested in Spark 3.1 should be primarily responsible for keeping it updated. Backporting patches is up to the contributor. The only concern I have about keeping Hive 3.1 is whether there are important bugs or security issues that are not getting backporte

Re: [DISCUSS] Switch to JDK 11 for releases?

Looks like Hive isn't quite done migrating to Java 11: https://issues.apache.org/jira/browse/HIVE-22415 I'm not sure whether that's still a problem, but we currently don't build Hive 3 support unless we're using Java 8. That makes me think that dropping JDK 8 support would probably also make it a

Re: [DISCUSS] Spark 3.1 support?

We backported only a small number of changes to 3.1, compared to 3.2. At this point, they also diverged quite a bit so doing those backports is hard. When we discussed how to support multiple engine versions, the community initially agreed that it’s optional for authors to cherry-pick changes in

Re: [DISCUSS] Switch to JDK 11 for releases?

I believe that Hive support still requires java 8 -Dan On Fri, Apr 21, 2023 at 5:02 PM Anton Okolnychyi wrote: > Sorry, I wasn’t clear that I also imply dropping JDK 8 (unless there is a > good reason to keep it?). > > - Anton > > On Apr 21, 2023, at 4:59 P

Re: [DISCUSS] Switch to JDK 11 for releases?

Sorry, I wasn’t clear that I also imply dropping JDK 8 (unless there is a good reason to keep it?). - Anton > On Apr 21, 2023, at 4:59 PM, Ryan Blue wrote: > > Would we also drop support for JDK 8? > > On Fri, Apr 21, 2023 at 4:58 PM Anton Okolnychyi > wrote: > Following up on the discussio

Re: [DISCUSS] Switch to JDK 11 for releases?

Would we also drop support for JDK 8? On Fri, Apr 21, 2023 at 4:58 PM Anton Okolnychyi wrote: > Following up on the discussion in the Spark 2.4 thread, shall we move to > JDK 11 for releases as Spark 2.4 support has been dropped? > > - Anton -- Ryan Blue Tabular

[DISCUSS] Switch to JDK 11 for releases?

Following up on the discussion in the Spark 2.4 thread, shall we move to JDK 11 for releases as Spark 2.4 support has been dropped? - Anton

Re: [DISCUSS] Spark 3.1 support?

Good question about backports. Walaa and Edgar, are you backporting fixes to 3.1? It makes sense to have a place to collaborate, but only if people are actively keeping them updated. On Fri, Apr 21, 2023 at 3:54 PM Steven Wu wrote: > For the 3.1 activities that Ryan linked, 3.1 are updated proba

Re: [DISCUSS] Spark 3.1 support?

For the 3.1 activities that Ryan linked, 3.1 are updated probably for the requirement of backporting (keeping 3.1, 3.2, 3.3 in sync). It is the adopted policy. Not sure if it is an indication that people are actively collaborating on 3.1. As Anton was saying, backporting/syncing 4 versions (3.1, 3

Re: [DISCUSS] Spark 3.1 support?

If it is being used by folks in the community, let’s keep it for now. That said, let’s come up with a strategy on when to eventually drop it as the list cannot grow indefinitely. Our initial agreement was to keep last 3 (except Spark LTS versions), which worked well for 18 months of support prom

Re: [DISCUSS] Spark 3.1 support?

According to Spark docs, a minor release will be supported for 18 months and 3.1 was released 2021-03-02, more than 2 years ago. I don't think we should expect any further updates from the Spark community for the 3.1 line. I'm also not sure that there is a problem continuing to release Iceberg's m

Re: [DISCUSS] Spark 3.1 support?

Airbnb is also still on Spark 3.1 and I echo some of Walaa's comments. Cheers, On Thu, Apr 20, 2023 at 8:14 PM Walaa Eldin Moustafa wrote: > LinkedIn is still on Spark 3.1. I am guessing a number of other companies > could be in the same boat. I feel the argument for Spark 2.4 is different > fr