Spring cleaning! I checked which versions of Spark the cloud vendors are supporting. Both AWS and GCP are already on 3.3. However, for Azure <https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-5x-component-versioning>, Spark 3.3 is in preview and is still on 3.1.3. They are planning to upgrade to Spark 3.2.0 <https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-5x-component-versioning#upcoming-version-upgrades> and I think we're fine for the next release of Iceberg.
Kind regards, Fokko Op do 20 apr 2023 om 22:22 schreef Anton Okolnychyi <aokolnyc...@apple.com.invalid>: > Since there are no objections and it is in line with what we planned > initially, I created a PR to drop 3.1. > > https://github.com/apache/iceberg/pull/7390 > > - Anton > > On Apr 19, 2023, at 11:05 AM, Ryan Blue <b...@tabular.io> wrote: > > +1 > > As we said in the 2.4 discussion, the format itself should provide forward > compatibility with tables and it is more clear that we aren't adding new > features if you have to use older versions for Spark 3.1. > > On Wed, Apr 19, 2023 at 10:08 AM Anton Okolnychyi < > aokolnyc...@apple.com.invalid> wrote: > >> Hey folks, >> >> What does everybody think about Spark 3.1 support after we add Spark 3.4 >> support? Our initial plan was to release jars for the last 3 versions. Are >> there any blockers for dropping 3.1? >> >> - Anton > > > > -- > Ryan Blue > Tabular > > >
