ramyadass commented on issue #864:
URL:
https://github.com/apache/datafusion-comet/issues/864#issuecomment-2579003972
@nblagodarnyi ,
I've tried to use Comet with Spark using two different commands, but I'm
encountering the same error in both cases.
1. Using local jar:
spark-shell --jars
/home/hadoop/.ivy2/jars/comet-spark-spark3.5_2.12-0.4.0.jar --conf
spark.driver.extraClassPath=/home/hadoop/.ivy2/jars/comet-spark-spark3.5_2.12-0.4.0.jar
--conf
spark.executor.extraClassPath=/home/hadoop/.ivy2/jars/comet-spark-spark3.5_2.12-0.4.0.jar
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
2. Using HDFS jar:
spark-shell --jars
hdfs:///home/hadoop/libraries/comet-spark-spark3.5_2.12-0.4.0.jar --conf
spark.driver.extraClassPath=/home/hadoop/.ivy2/jars/comet-spark-spark3.5_2.12-0.4.0.jar
--conf
spark.executor.extraClassPath=/home/hadoop/.ivy2/jars/comet-spark-spark3.5_2.12-0.4.0.jar
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
Error received in both cases:
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
I've verified that the jar file contains the specified class.
1. Are there any additional steps or pre-configurations I might be missing?
2. Do you have any suggestions for resolving this ClassNotFoundException?
Any insights would be greatly appreciated. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
