+1, Spark 2.4 is very out of sync with current developments as noted above. It's almost impossible for us to get any newer features to be compatible with it.
On Fri, Apr 14, 2023 at 4:52 PM Anjali Norwood <anorw...@netflix.com.invalid> wrote: > Hi Fokko, Ryan, > > Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are > actively migrating to Spark-3.x and Iceberg 1.1 (or later). I do not > anticipate us using Spark-2.4.4 with newer versions of Iceberg (>0.9). > If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that should > be ok. > @John Zhuge <jzh...@netflix.com> can you please chime in? > > thanks, > Anjali > > > On Fri, Apr 14, 2023 at 10:56 AM Ryan Blue <b...@tabular.io> wrote: > >> Overall I'm +1, but could be convinced otherwise. >> >> Spark 2.4 is old and doesn't really function properly because the Spark >> Catalog API was missing at the time. And people can still use older >> versions of Iceberg that support Spark 2.4 if they need it because the >> Iceberg spec guarantees forward compatibility. >> >> That said, I'd love to hear from more people on this. I think it would be >> great to drop support, but I don't know how many people still use it. Is >> upgrading Hadoop a good reason to drop support for an engine? Hadoop seems >> like a minor concern to me unless it is blocking something. >> >> Ryan >> >> On Thu, Apr 13, 2023 at 12:54 PM Jack Ye <yezhao...@gmail.com> wrote: >> >>> +1 for dropping 2.4 support >>> >>> Best, >>> Jack Ye >>> >>> On Thu, Apr 13, 2023 at 10:59 AM Fokko Driesprong <fo...@apache.org> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm working on moving to Hadoop 3.x >>>> <https://github.com/apache/iceberg/pull/7114>, and one thing is that >>>> it seems to be incompatible with Spark 2.4. I wanted to ask if people are >>>> still on Spark 2.4 and what we think of dropping the support. The last >>>> release of Spark 2.4.8 was on 2021-05-17 and it also looks like the 2.4 >>>> branch on the Spark Github repository is stale, so I don't expect any >>>> further releases. >>>> >>>> Before creating a PR I would like to check on the mail-list if anyone >>>> has any objections. If so, please let us know. >>>> >>>> Thanks, >>>> Fokko Driesprong >>>> >>> >> >> -- >> Ryan Blue >> Tabular >> >
