I look into the "Environment" in both modes. yarn-client: spark.jars local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar,file:/home/xxx/my-app.jar yarn-cluster: spark.yarn.secondary.jars local:/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar I wonder why htrace exists in "spark.yarn.secondary.jars" but still not found in URLClassLoader.
I tried both "local" and "file" mode for the jar, still the same error. 2015-05-14 11:37 GMT+08:00 Fengyun RAO <raofeng...@gmail.com>: > Hadoop version: CDH 5.4. > > We need to connect to HBase, thus need extra > "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" > dependency. > > It works in yarn-client mode: > "spark-submit --class xxx.xxx.MyApp --master yarn-client --num-executors > 10 --executor-memory 10g --jars > /opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar > my-app.jar /input /output" > > However, if we change "yarn-client" to "yarn-cluster', it throws an > ClassNotFoundException (actually the class exists in > htrace-core-3.1.0-incubating.jar): > > Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218) > at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481) > at > org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65) > at > org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:850) > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635) > ... 21 more > Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > > Why --jars doesn't work in yarn-cluster mode? How to add extra dependency in > "yarn-cluster" mode? > > >
