pan3793 commented on issue #7673: URL: https://github.com/apache/kyuubi/issues/7673#issuecomment-5497828751
it's a kyuubi side bug, I had [a draft PR](https://github.com/apache/kyuubi/pull/7064) that tries to fix it, but didn't find time to complete it. the key point is, spark's default catalog `spark_catalog` (called session catalog) has a special internal impl that does not fully follow the DSv2 API, when set `spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog`, spark should use the configured `o.a.iceberg.spark.SparkSessionCatalog` to resolve tables, but our current impl prefer to use the built-in v1 session catalog for `spark_catalog` to resolve table on the GetTables API, so the iceberg fails to load by its catalog and is treated at a normal Hive SerDe table, then exception happens. and your new setup, which drops `iceberg-hive-runtime-<version>.jar` spark classpath, is absolutely right, it's strongly discouraged to add this jar to spark classpath, when ClassNotFound exception that points to a class that exists in the `iceberg-hive-runtime-<version>.jar`, it means there is a bug, blindly adding the `iceberg-hive-runtime-<version>.jar` likely masks the real issue, and in worst case, your spark read and write the iceberg table through Hive SerDe approach, without even basic performance optimization like partition pruning, it's horrible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
