Probably because your JAR file requires other JARs which you didn't supply. If you specify a package, it reads metadata like a pom.xml file to understand what other dependent JARs also need to be loaded.
On Tue, Oct 20, 2020 at 10:50 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi, > > I have a scenario that I use in Spark submit as follows: > > spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars > /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar, > */home/hduser/jars/spark-bigquery_2.11-0.2.6.jar* > > As you can see the jar files needed are added. > > > This comes back with error message as below > > > Creating model test.weights_MODEL > > java.lang.NoClassDefFoundError: > com/google/api/client/http/HttpRequestInitializer > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.bq$lzycompute(BigQuerySQLContext.scala:19) > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.bq(BigQuerySQLContext.scala:19) > > at > com.samelamin.spark.bigquery.BigQuerySQLContext.runDMLQuery(BigQuerySQLContext.scala:105) > > ... 76 elided > > Caused by: java.lang.ClassNotFoundException: > com.google.api.client.http.HttpRequestInitializer > > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > So there is an issue with finding the class, although the jar file used > > > /home/hduser/jars/spark-bigquery_2.11-0.2.6.jar > > has it. > > > Now if *I remove the above jar file and replace it with the same version > but package* it works! > > > spark-submit --driver-class-path /home/hduser/jars/ddhybrid.jar --jars > /home/hduser/jars/spark-bigquery-latest.jar,/home/hduser/jars/ddhybrid.jar > *-**-packages com.github.samelamin:spark-bigquery_2.11:0.2.6* > > > I have read the write-ups about packages searching the maven > libraries etc. Not convinced why using the package should make so much > difference between a failure and success. In other words, when to use a > package rather than a jar. > > > Any ideas will be appreciated. > > > Thanks > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > >
