Re: Help needed with Iceberg on Databricks platform

2023-04-28 Thread Anton Okolnychyi
It looks like you are still pulling in some 0.11 version classes. That version is very old and may not contain `ResolveProcedures`. I assume you have classpath first flag enabled? - Anton > On Apr 28, 2023, at 3:18 PM, Pani Dhakshnamurthy wrote: > > Hi John, > I tried using both the "set spar

Re: Help needed with Iceberg on Databricks platform

2023-04-28 Thread Pani Dhakshnamurthy
Hi John, I tried using both the "set spark.sql.extensions" command and "spark.conf.get("spark.sql.extensions")" method, but both returned "undefined." To resolve this issue, I attempted to set the following values: spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtens

Re: Help needed with Iceberg on Databricks platform

2023-04-28 Thread John Zhuge
Have you tried this? 1. Copy the value of `spark.sql.extensions` (a comma separated list) from an existing Spark job on Databricks 2. Append `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the comma separated list 3. Submit a new job with `spark.sql.exten

Help needed with Iceberg on Databricks platform

2023-04-28 Thread Pani Dhakshnamurthy
Hi all, I am currently attempting to make Iceberg work on the Databricks platform. Without adding IcebergSparkSessionExtensions, I am able to perform both data and metadata reads. However, when I add IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws the following error: C