Hi, I have a scenario that I use in Spark submit as follows:
spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar, */home/hduser/jars/spark-bigquery_2.11-0.2.6.jar* As you can see the jar files needed are added. This comes back with error message as below Creating model test.weights_MODEL java.lang.NoClassDefFoundError: com/google/api/client/http/HttpRequestInitializer at com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) at com.samelamin.spark.bigquery.BigQuerySQLContext.runDMLQuery(BigQuerySQLContext.scala:105) ... 76 elided Caused by: java.lang.ClassNotFoundException: com.google.api.client.http.HttpRequestInitializer at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) So there is an issue with finding the class, although the jar file used /home/hduser/jars/spark-bigquery_2.11-0.2.6.jar has it. Now if *I remove the above jar file and replace it with the same version but package* it works! spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar *-**-packages com.github.samelamin:spark-bigquery_2.11:0.2.6* I have read the write-ups about packages searching the maven libraries etc. Not convinced why using the package should make so much difference between a failure and success. In other words, when to use a package rather than a jar. Any ideas will be appreciated. Thanks *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
