I initially supported not renaming for the reason that Jeff raised, but now I'm more convinced by Kyle's argument. This is confusing and it isn't that big of a problem to use a different Jar. +1 to renaming.
On Sun, Feb 20, 2022 at 10:57 PM Yufei Gu <flyrain...@gmail.com> wrote: > Agreed with Kyle. An artifact name of Spark3.0 like > iceberg-spark-runtime-3.0_2.12-0.13.1.jar is more accurate and > consistent, less confusing for users. > > On Sun, Feb 20, 2022 at 10:47 PM Kyle Bendickson <k...@tabular.io> wrote: > >> Thanks for bringing this up Jeff! >> >> Normally I agree, it’s not a good practice to change artifact name. >> However, in this case, the artifact has changed already. The >> “spark3-runtime” used to be for all versions of Spark 3 (at the time Spark >> 3.0 and 3.1). It no longer is, as it’s only tested / used with Spark 3.0. >> >> I encounter many users who have upgraded to newer versions of Spark, but >> have not upgraded the artifact to the newly versioned by Spark name system >> as “spark3-runtime” sounds like it encompasses all versions. And they >> encounter subtle bugs and it’s not a great user experience to solve >> upgrading that way. >> >> These users are, however, updating the Iceberg artifact to the new >> versions. >> >> So I think in this case, breaking naming has benefits. As users who go to >> upgrade when new Iceberg version are released, and their dependency is not >> found, they will hopefully check maven and see the new naming convention / >> artifacts. >> >> So I support option 2 also, with naming with Spark and Scala versions. >> Otherwise, we continue to see people using the old “spark3-runtime” as they >> upgrade Spark versions and encounter subtle errors (class not found, wrong >> type signatures due to version mismatch). >> >> Users eventually have to upgrade their pom if / when they upgrade Spark, >> due to incompatibility. This way at least, breaking will be loud as there’s >> won’t be a new Iceberg version, >> >> Is it possible to mark to the old spark3-runtime / spark-runtime as >> deprecated or otherwise point to the new artifacts in Maven? >> >> - Kyle >> >> On Sun, Feb 20, 2022 at 9:41 PM Jeff Zhang <zjf...@gmail.com> wrote: >> >>> I don't think it is best practice to just change the artifact name of >>> published jars. Unless we publish a new version with the new naming >>> convention. >>> >>> On Mon, Feb 21, 2022 at 12:36 PM Jack Ye <yezhao...@gmail.com> wrote: >>> >>>> I think option 2 is ideal, but I don't know if there is any hard >>>> requirement from ASF/Maven Central side for us to keep backwards >>>> compatibility of package names published in maven. If there is a >>>> requirement then we cannot change it. >>>> >>>> As a mitigation, I stated in >>>> https://iceberg.apache.org/multi-engine-support that Spark 2.4 and 3.0 >>>> jar names do not follow the naming convention of newer versions for >>>> backwards compatibility. >>>> >>>> Best, >>>> Jack Ye >>>> >>>> On Sun, Feb 20, 2022 at 7:03 PM OpenInx <open...@gmail.com> wrote: >>>> >>>>> Hi everyone >>>>> >>>>> The current spark2.4, spark3.0 have the following unaligned runtime >>>>> artifact names: >>>>> >>>>> # Spark 2.4 >>>>> iceberg-spark-runtime-0.13.1.jar >>>>> # Spark 3.0 >>>>> iceberg-spark3-runtime-0.13.1.jar >>>>> # Spark 3.1 >>>>> iceberg-spark-runtime-3.1_2.12-0.13.1.jar >>>>> # Spark 3.2 >>>>> iceberg-spark-runtime-3.2_2.12-0.13.1.jar >>>>> >>>>> From the spark 3.1 and spark 3.2's runtime artifact names, we can >>>>> easily recognize: >>>>> 1. What's the spark major version that the runtime jar is attached to >>>>> 2. What's the spark scala version that the runtime jar is compiled with >>>>> >>>>> But for spark 3.0 and spark 2.4, it's not easy to understand what's >>>>> the above information. I think we kept those legacy names because they >>>>> were introduced in older iceberg releases and we wanted to avoid changing >>>>> the modules that users depend on and opted not to rename, but they are >>>>> indeed causing confusion for the new community users. >>>>> >>>>> In general, we have two options: >>>>> >>>>> Option#1: keep the current artifact names, that mean spark 2.4 & >>>>> spark 3.0 will always use the iceberg-spark-runtime-<iceberg-version>.jar >>>>> and iceberg-spark3-runtime-<iceberg-version>.jar until them get retired in >>>>> the apache iceberg official repo. >>>>> Option#2: Change the spark2.4 & spark3.0's artifact names to the >>>>> generic name format: >>>>> iceberg-spark-runtime-<spark-major.minor>_<scala-version>-<iceberg-version>.jar. >>>>> It makes sharing all the consistent name format between all the spark >>>>> versions. >>>>> >>>>> Personally, I'd prefer option#2 because that looks more friendly for >>>>> new community users (although it will require the old users to change >>>>> their >>>>> pom.xml to the new version). >>>>> >>>>> What is your preference ? >>>>> >>>>> Reference: >>>>> 1. Created a PR to change the artifact names and we had few >>>>> discussions there. https://github.com/apache/iceberg/pull/4158 >>>>> 2. >>>>> https://github.com/apache/iceberg-docs/pull/27#discussion_r800297155 >>>>> >>>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> -- > Best, > > Yufei > > `This is not a contribution` > -- Ryan Blue Tabular
