karuppayya opened a new pull request, #3738:
URL: https://github.com/apache/datafusion-comet/pull/3738
## Which issue does this PR close?
Closes https://github.com/apache/datafusion-comet/issues/3737
## Rationale for this change
Comet uses `Class.forName(className)` to reflectively load Iceberg classes.
The single-argument `Class.forName` uses the classloader of the **calling
class**, not the thread context classloader.
When the Comet JAR is placed in Spark's `jars/` folder, it is loaded by the
system classloader.
However, Iceberg classes may be bundled inside the user's application JAR(as
uber jar), which is loaded by Spark's [application
classloader](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L939-L950)
.
Since parent classloaders cannot see classes loaded by child classloaders,
`Class.forName` from Comet's code fails with `ClassNotFoundException`.
([`SparkClassUtils.classForName`](https://github.com/apache/spark/blob/master/common/utils/src/main/scala/org/apache/spark/util/SparkClassUtils.scala#L37-L48)
I guess does the same)
## What changes are included in this PR?
- Adds `IcebergReflection.loadClass()` helper that resolves classes via
the thread context classloader (falling back to the default if null).
- Replaces all bare `Class.forName` calls in `IcebergReflection` and
`CometIcebergNativeScan` with the new helper method.
## How are these changes tested?
Verified in a deployment where the Comet JAR is in Spark's `jars/` folder
and Iceberg is bundled in the application uber JAR and ran few SQLs which
involved reading from Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
