Spark Unit tests - RDDBlockId not found

I am running unit tests on Spark 1.3.1 with sbt test and besides the unit tests being incredibly slow I keep running into java.lang.ClassNotFoundException: org.apache.spark.storage.RDDBlockId issues. Usually this means a dependency issue, but I wouldn't know from where... Any help is greatly appre

Re: GroupBy on RDD returns empty collection

I just ran the same app with limited data on my personal machine - no error. Seems to be a mesos issue. Will investigate further. If anyone knows anything, let me know :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/GroupBy-on-RDD-returns-empty-collection

GroupBy on RDD returns empty collection

I noticed that my spark jobs suddenly return empty data and tried to find out why. It seems as if a groupBy operation is the cause of it. When I run val original:RDD[Data] val x = original.cache().groupBy(x=>(x.first,x.last,x.date)) and then try println(s"${x.first()}") I get an Exception in th

Re: equivalent to sql in

This is more a scala specific question. I would look at the List contains implementation -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/equivalent-to-sql-in-tp20599p20600.html Sent from the Apache Spark User List mailing list archive at Nabble.com. ---

Writing and reading file faster than memory option

I am facing a somewhat confusing problem: My spark app reads data from a database, calculates certain values and then runs a shortest path Pregel operation on them. If I save the RDD to disk and then read the information out again, my app runs between 30-50% faster than keeping it in memory, plus

Problems with ZooKeeper and key canceled

I have a spark cluster on mesos and when I run long running GraphX processing I receive a lot of the following two errors and one by one my slaves stop doing any work for the process until its idle. Any idea what is happening? First type of error message: INFO SendingConnection: Initiating connec

Re: Spark Worker crashing and Master not seeing recovered worker

This is still happening to me on mesos. Any workarounds? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Worker-crashing-and-Master-not-seeing-recovered-worker-tp2312p16506.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: "Initial job has not accepted any resources" when launching SparkPi example on a worker.

Besides the host1 question what can also happen is that you give the worker more memory than available (try a value 1G below the memory available just to be sure for example) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Initial-job-has-not-accepted-any-re

RE: Spark on Mesos Issue - Do I need to install Spark on Mesos slaves

I have actually had the same problem. spark.executor.uri on HDFS did not work so I had to put it in a local folder -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Mesos-Issue-Do-I-need-to-install-Spark-on-Mesos-slaves-tp16129p16165.html Sent from th