I think it should be minimally difficult to switch this around on the Iceberg side, we only have to move the initialize code out and duplicate it. Not a huge cost
On Sun, Sep 22, 2024 at 11:39 PM Wenchen Fan <cloud0...@gmail.com> wrote: > It's a buggy behavior that a custom v2 catalog (without extending > DelegatingCatalogExtension) expects Spark to still use the v1 DDL commands > to operate on the tables inside it. This is also why the third-party > catalogs (e.g. Unity Catalog and Apache Polaris) can not be used to > overwrite `spark_catalog` if people still want to use the Spark built-in > file sources. > > Technically, I think it's wrong for a third-party catalog to rely on > Spark's session catalog without extending `DelegatingCatalogExtension`, as > it confuses Spark. If it has its own metastore, then it shouldn't delegate > requests to the Spark session catalog and use v1 DDL commands which only > work with the Spark session catalog. Otherwise, it should extend > `DelegatingCatalogExtension` to indicate it. > > On Mon, Sep 23, 2024 at 11:19 AM Manu Zhang <owenzhang1...@gmail.com> > wrote: > >> Hi Iceberg and Spark community, >> >> I'd like to bring your attention to a recent change[1] in Spark 3.5.3 >> that effectively breaks Iceberg's SparkSessionCatalog[2] and blocks Iceberg >> upgrading to Spark 3.5.3[3]. >> >> SparkSessionCatalog, as a customized Spark V2 session catalog, >> supports creating a V1 table with V1 command. That's no longer allowed >> after the change unless it extends DelegatingCatalogExtension. It is not >> minor work since SparkSessionCatalog already extends a base class[4]. >> >> To resolve this issue, we have to make changes to public interfaces at >> either Spark or Iceberg side. IMHO, it doesn't make sense for a downstream >> project to refactor its interfaces when bumping up a maintenance version of >> Spark. WDYT? >> >> >> 1. https://github.com/apache/spark/pull/47724 >> 2. >> https://iceberg.apache.org/docs/nightly/spark-configuration/#replacing-the-session-catalog >> 3. https://github.com/apache/iceberg/pull/11160 >> <https://github.com/apache/iceberg/pull/11160> >> 4. >> https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java >> >> Thanks, >> Manu >> >>
