Oops I think that should fix it. I am going to try it now.. Great catch! I feel like an idiot.
On Fri, Feb 17, 2017 at 10:02 AM, Russell Spitzer <russell.spit...@gmail.com > wrote: > Great catch Anastasios! > > On Fri, Feb 17, 2017 at 9:59 AM Anastasios Zouzias <zouz...@gmail.com> > wrote: > >> Hey, >> >> Can you try with the 2.11 spark-cassandra-connector? You just reported >> that you use spark-cassandra-connector*_2.10*-2.0.0-RC1.jar >> >> Best, >> Anastasios >> >> On Fri, Feb 17, 2017 at 6:40 PM, kant kodali <kanth...@gmail.com> wrote: >> >> Hi, >> >> >> val df = spark.read.format("org.apache.spark.sql.cassandra").options(Map( >> "table" -> "hello", "keyspace" -> "test" )).load() >> >> This line works fine. I can see it actually pulled the table schema from >> cassandra. however when I do >> >> df.count I get the error below. >> >> >> I am using the following jars. >> >> spark version 2.0.2 >> >> spark-sql_2.11-2.0.2.jar >> >> spark-cassandra-connector_2.10-2.0.0-RC1.jar >> >> Java version 8 >> >> scala version 2.11.8 >> >> >> >> java.lang.NoClassDefFoundError: scala/runtime/ >> AbstractPartialFunction$mcJL$sp >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) >> >> at java.security.SecureClassLoader.defineClass( >> SecureClassLoader.java:142) >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.lang.ClassNotFoundException: scala.runtime. >> AbstractPartialFunction$mcJL$sp >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >> ... 35 more >> >> 17/02/17 17:35:33 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID >> 0) >> >> java.lang.NoClassDefFoundError: com/datastax/spark/connector/ >> rdd/CassandraLimit$$anonfun$limitForIterator$1 >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 17/02/17 17:35:33 ERROR Executor: Exception in task 2.0 in stage 0.0 (TID >> 2) >> >> java.lang.NoClassDefFoundError: com/datastax/spark/connector/ >> rdd/CassandraLimit$$anonfun$limitForIterator$1 >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 17/02/17 17:35:33 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID >> 3) >> >> java.lang.NoClassDefFoundError: com/datastax/spark/connector/ >> rdd/CassandraLimit$$anonfun$limitForIterator$1 >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 17/02/17 17:35:33 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, >> localhost): java.lang.NoClassDefFoundError: scala/runtime/ >> AbstractPartialFunction$mcJL$sp >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) >> >> at java.security.SecureClassLoader.defineClass( >> SecureClassLoader.java:142) >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.lang.ClassNotFoundException: scala.runtime. >> AbstractPartialFunction$mcJL$sp >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> >> ... 35 more >> >> >> 17/02/17 17:35:33 ERROR TaskSetManager: Task 1 in stage 0.0 failed 1 >> times; aborting job >> >> 17/02/17 17:35:33 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, >> localhost): java.lang.NoClassDefFoundError: com/datastax/spark/connector/ >> rdd/CassandraLimit$$anonfun$limitForIterator$1 >> >> at com.datastax.spark.connector.rdd.CassandraLimit$.limitForIterator( >> CassandraLimit.scala:21) >> >> at com.datastax.spark.connector.rdd.CassandraTableScanRDD. >> compute(CassandraTableScanRDD.scala:367) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.rdd.MapPartitionsRDD.compute( >> MapPartitionsRDD.scala:38) >> >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) >> >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:79) >> >> at org.apache.spark.scheduler.ShuffleMapTask.runTask( >> ShuffleMapTask.scala:47) >> >> at org.apache.spark.scheduler.Task.run(Task.scala:86) >> >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) >> >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1142) >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> >> -- >> -- Anastasios Zouzias >> <a...@zurich.ibm.com> >> >