Re: [DISCUSS] Dropping Spark 2.4 support

2023-04-14 Thread John Zhuge
+1 on removing 2.4 support On Fri, Apr 14, 2023 at 5:31 PM John Zhuge wrote: > Netflix internal Spark 2.4 is different from OSS. It is closer to OSS 3.0 > or 3.1 because it has DataSourceV2 and catalog support. So we don't rely on > Iceberg Spark 2.4 code. > > On Fri, Apr 14, 2023 at 3:12 PM Rus

Re: [DISCUSS] Dropping Spark 2.4 support

2023-04-14 Thread John Zhuge
Netflix internal Spark 2.4 is different from OSS. It is closer to OSS 3.0 or 3.1 because it has DataSourceV2 and catalog support. So we don't rely on Iceberg Spark 2.4 code. On Fri, Apr 14, 2023 at 3:12 PM Russell Spitzer wrote: > +1, Spark 2.4 is very out of sync with current developments as no

Re: [DISCUSS] Dropping Spark 2.4 support

2023-04-14 Thread Russell Spitzer
+1, Spark 2.4 is very out of sync with current developments as noted above. It's almost impossible for us to get any newer features to be compatible with it. On Fri, Apr 14, 2023 at 4:52 PM Anjali Norwood wrote: > Hi Fokko, Ryan, > > Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are > act

Re: [DISCUSS] Dropping Spark 2.4 support

2023-04-14 Thread Anjali Norwood
Hi Fokko, Ryan, Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are actively migrating to Spark-3.x and Iceberg 1.1 (or later). I do not anticipate us using Spark-2.4.4 with newer versions of Iceberg (>0.9). If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that should be ok. @Jo

Re: [DISCUSS] Dropping Spark 2.4 support

2023-04-14 Thread Ryan Blue
Overall I'm +1, but could be convinced otherwise. Spark 2.4 is old and doesn't really function properly because the Spark Catalog API was missing at the time. And people can still use older versions of Iceberg that support Spark 2.4 if they need it because the Iceberg spec guarantees forward compa