Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread russell . spitzer
If you are already back-porting patches to a different branch which isn’t receiving other fixes anyway, why does it help to keep 3.1 support in master? If we kept it there, we would be doing so for users who want a 1.2+ Iceberg version with 3.1 support. That doesn’t sound like your use case. I’m no

Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread Manu Zhang
Hi Russel, What do you mean by "keep these changes in master"? Can you elaborate? As for Iceberg, we back-port spark/v3.1 patches from master branch. On Sun, Apr 23, 2023 at 10:04 AM wrote: > If you are on forked 0.13 is it important to keep these changes in master? > > Sent from my iPhone > >

Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread russell . spitzer
If you are on forked 0.13 is it important to keep these changes in master?Sent from my iPhoneOn Apr 22, 2023, at 8:42 PM, Manu Zhang wrote:I'd like to share our maintenance strategy and history at eBay.We are now on forked versions of Iceberg 0.13.1 and Spark 3.1.1. For Spark, We started to evalu

Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread Manu Zhang
I'd like to share our maintenance strategy and history at eBay. We are now on forked versions of Iceberg 0.13.1 and Spark 3.1.1. For Spark, We started to evaluate upgrading to 3.1.1 from 2.3/2.4 in H2, 2021 since it was the latest and most stable version then. After migrating internal changes and

Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread Ryan Blue
+1 for marking 3.1 deprecated. On Sat, Apr 22, 2023 at 10:20 AM Jack Ye wrote: > Here was the original lifecycle of engine version support guideline we > came up with: > https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status > > I think we can at least mark 3.1

Re: [DISCUSS] Switch to JDK 11 for releases?

2023-04-22 Thread Ryan Blue
I think in order to publish multiple versions we would need to have different artifact names, like Scala uses (e.g. _2.12). It probably also wouldn't help. If we have to remain compatible with JDK 8, then publishing some artifacts for JDK 11 would still mean only using JDK 8 features. The source v

Re: [DISCUSS] Spark 3.1 support?

2023-04-22 Thread Jack Ye
Here was the original lifecycle of engine version support guideline we came up with: https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status I think we can at least mark 3.1 support as deprecated, which matches the situation here that "People who are still interest

Re: [DISCUSS] Switch to JDK 11 for releases?

2023-04-22 Thread Jack Ye
Would it be an option to use --release flag to control the release target version, and publish 2 versions of the library to Maven, 1 for JDK8 and 1 for JDK11? Jack On Fri, Apr 21, 2023 at 5:17 PM Ryan Blue wrote: > Looks like Hive isn't quite done migrating to Java 11: > https://issues.apache.o