Not sure if this problem reached the Spark guys because it shows in Nabble that "This post has NOT been accepted by the mailing list yet".
http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFound-for-user-class-in-uber-jar-td10613.html#a11902 I'm resubmitting. Greetings, I'm currently building a "fat" or "uber" jar with dependencies using maven that a docker-ized Spark cluster (1 master 3 workers, version 1.0.0, scala 2.10.4) points to locally on the same VM. It seems that sometimes a particular class is found, and things are fine, and other times it is not. Doing a find on the jar affirms that it is actually there. I `setJars` with JavaStreamingContext.jarOfClass(the main class). I cannot say I know much about how the ClassPath mechanisms of Spark so I appreciate any and all suggestions to find out what exactly is happening. The exception is as follows: 14/07/24 18:48:52 INFO Executor: Sending result for 139 directly to driver 14/07/24 18:48:52 INFO Executor: Finished task ID 139 14/07/24 18:48:56 WARN BlockManager: Putting block input-0-1406227713800 failed 14/07/24 18:48:56 ERROR BlockManagerWorker: Exception handling buffer message java.lang.ClassNotFoundException: com.cjm5325.MyProject.MyClass at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:60) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:125) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.util.NextIterator.foreach(NextIterator.scala:21) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at org.apache.spark.storage.MemoryStore.putBytes(MemoryStore.scala:59) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:666) at org.apache.spark.storage.BlockManager.putBytes(BlockManager.scala:587) at org.apache.spark.storage.BlockManagerWorker.putBlock(BlockManagerWorker.scala:82) at org.apache.spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:63) at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44) at org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at org.apache.spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:28) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at org.apache.spark.storage.BlockMessageArray.map(BlockMessageArray.scala:28) at org.apache.spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:44) at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34) at org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34) at org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:661) at org.apache.spark.network.ConnectionManager$$anon$9.run(ConnectionManager.scala:503) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Some classpath information: Classpath Entries Resource (Source) /opt/spark-1.0.0/conf (System Classpath) /opt/spark-1.0.0/lib/spark-assembly-1.0.0-hadoop1.0.4.jar (System Classpath) /opt/spark-1.0.0/work/driver-20140724160201-0002/my-jar-with-dependencies.jar (System Classpath) http://spark.filesystem.uri/jars/my-jar-with-dependencies.jar (Added by user) If more information is needed, please let me know and I will be glad to provide it. I have searched for similar issues with mild success, but I cannot reason why sometimes the class would be loaded and others not. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFound-exception-on-class-in-uber-jar-tp11903.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
