The issue seems to be with primordial class loader. I cannot load the drivers to all the nodes at the same location but have loaded the jars to HDFS. I have tried SPARK_YARN_DIST_FILES as well as SPARK_CLASSPATH on the edge node with no luck. Is there another way to load these jars through primordial class loader in YARN CLUSTER mode ? or do I have to add them inside SPARK assembly jar?
Thank You for all the help. On Wed, May 31, 2017 at 10:51 AM, morfious902002 <anubha...@gmail.com> wrote: > Hi, > I am trying to create a Dataframe by querying Impala Table. It works fine > in > my local environment but when I try to run it in cluster I either get > > Error:java.lang.ClassNotFoundException: com.cloudera.impala.jdbc41.Driver > > or > > No Suitable Driver found. > > Can someone help me or direct me to how I can accomplish this? > > I am using Spark 1.6.1. Here is my command (No Suitable Driver found error) > :- > '/appserver/spark/spark-1.6.1-bin-hadoop2.6/bin/spark-submit' '--master' > 'yarn' '--deploy-mode' 'cluster' '--name' 'Livy' '--jars' > "hdfs:///user/lib/ImpalaJDBC41.jar,hdfs:///user/lib/TCLIServiceClient.jar, > hdfs:///user/lib/libfb303-0.9.0.jar,hdfs:///user/lib/ > libthrift-0.9.0.jar,hdfs:///user/lib/hive_metastore.jar, > hdfs:///user/lib/hive_service.jar" > '--class' 'Main.class' '--driver-memory' '5G' '--driver-cores' '2' > '--executor-memory' '8G' '--driver-cores' '2' '--executor-cores' '3' > '--num-executors' '2' 'my.jar' 'arg' 'arg' 'arg' > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Creating-Dataframe-by-querying-Impala-tp28723.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
