Hi, I'm trying to run a phoenix spark job via spark cluster mode to a remote yarn cluster.
When I do a spark-submit, all jars under SPARK_HOME gets uploaded. I also need to point the remote hbase jar folder location and other dependencies for running the job. Going through the docs, I see setting spark.yarn.jars is one way of doing it. For spark.executor.extraClassPath, I see "Extra classpath entries to prepend to the classpath of executors. This exists primarily for backwards-compatibility with older versions of Spark. *Users typically should not need to set this option*." Similarly, spark.driver.extraClassPath works only for spark driver. For running the above job, I need all executors to know hbase and other dependencies path. So, is setting spark.yarn.jars the preferred way for submitting spark jobs remotely? Appreciate any input. Thanks!
