Missed to mention it above, but just to add, the error is coming from the driver. I tried using *--driver-class-path /path/to/my/jar* as well, but no luck.
Thanks! On Mon, Dec 12, 2022 at 4:21 PM Hariharan <hariharan...@gmail.com> wrote: > Hello folks, > > I have a spark app with a custom implementation of > *fs.s3a.s3.client.factory.impl* which is packaged into the same jar. > Output of *jar tf* > > *2620 Mon Dec 12 11:23:00 IST 2022 aws/utils/MyS3ClientFactory.class* > > However when I run the my spark app with spark-submit in cluster mode, it > fails with the following error: > > *java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: Class aws.utils.MyS3ClientFactory not > found* > > I tried: > 1. passing in the jar to the *--jars* option (with the local path) > 2. Passing in the jar to *spark.yarn.jars* option with an HDFS path > > but still the same error. > > Any suggestions on what I'm missing? > > Other pertinent details: > Spark version: 3.3.0 > Hadoop version: 3.3.4 > > Command used to run the app > */spark/bin/spark-submit --class MyMainClass --deploy-mode cluster > --master yarn --conf spark.executor.instances=6 /path/to/my/jar* > > TIA! >
