Hi,
We are testing spark-connect with iceberg.
We tried spark 3.5, iceberg 1.4.x versions (all of
iceberg-spark-runtime-3.5_2.12-1.4.x.jar)

with all the 1.4.x jars we are having following issue when running iceberg
queries from sparkSession created using spark-connect (--remote
"sc://remote-master-node")

org.apache.iceberg.spark.source.SerializableTableWithSize cannot be cast to
org.apache.iceberg.Table at
org.apache.iceberg.spark.source.SparkInputPartition.table(SparkInputPartition.java:88)
at
org.apache.iceberg.spark.source.BatchDataReader.<init>(BatchDataReader.java:50)
at
org.apache.iceberg.spark.source.SparkColumnarReaderFactory.createColumnarReader(SparkColumnarReaderFactory.java:52)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:79)
at
org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:63)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown
Source) at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithKeys_0$(Unknown
Source) at

Someone else has reported this issue on github as well:
https://github.com/apache/iceberg/issues/8978

It's currently working with spark 3.4 and iceberg 1.3 . However Ideally
it'd be nice to get it working with spark 3.5 as well as 3.5 has many
improvements in spark-connect.

Thanks
Nirav

Reply via email to