+1 on removing 2.4 support
On Fri, Apr 14, 2023 at 5:31 PM John Zhuge wrote:
> Netflix internal Spark 2.4 is different from OSS. It is closer to OSS 3.0
> or 3.1 because it has DataSourceV2 and catalog support. So we don't rely on
> Iceberg Spark 2.4 code.
>
> On Fri, Apr 14, 2023 at 3:12 PM Rus
Netflix internal Spark 2.4 is different from OSS. It is closer to OSS 3.0
or 3.1 because it has DataSourceV2 and catalog support. So we don't rely on
Iceberg Spark 2.4 code.
On Fri, Apr 14, 2023 at 3:12 PM Russell Spitzer
wrote:
> +1, Spark 2.4 is very out of sync with current developments as no
+1, Spark 2.4 is very out of sync with current developments as noted above.
It's almost impossible for us to get any newer features to be compatible
with it.
On Fri, Apr 14, 2023 at 4:52 PM Anjali Norwood
wrote:
> Hi Fokko, Ryan,
>
> Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are
> act
Hi Fokko, Ryan,
Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are actively migrating
to Spark-3.x and Iceberg 1.1 (or later). I do not anticipate us using
Spark-2.4.4 with newer versions of Iceberg (>0.9).
If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that should
be ok.
@Jo
Overall I'm +1, but could be convinced otherwise.
Spark 2.4 is old and doesn't really function properly because the Spark
Catalog API was missing at the time. And people can still use older
versions of Iceberg that support Spark 2.4 if they need it because the
Iceberg spec guarantees forward compa
