getting started writing unit tests for my app

Hi Folks, I'm writing a GraphX app and I need to do some test-driven development. I've got Spark running on our little cluster and have built and run some hello world apps, so that's all good. I've looked through the test source and found lots of helpful examples that use SharedSparkContext, and

GraphX pregel: getting the current iteration number

Hi Folks, I'm new to GraphX and Scala and my sendMsg function needs to index into an input list to my algorithm based on the pregel()() iteration number, but I don't see a way to access that. I see in https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/Preg

failing GraphX application ('GC overhead limit exceeded', 'Lost executor', 'Connection refused', etc.)

pe you can provide some advice on things to try next (e.g., configuration vars). My guess is the cluster is running out of memory, though I think it has adequate aggregate ram to handle this app. Thanks very much -- matt Matthew Cornell, Research Fellow, Computer Science Department, Umass

Re: failing GraphX application ('GC overhead limit exceeded', 'Lost executor', 'Connection refused', etc.)

nodes, not just the 0-3 one, so it's not node-specific. Any ideas about how to fix this? Thanks again, matt On Feb 12, 2015, at 10:46 AM, Matthew Cornell wrote: > Hi Folks, > > I'm running a five-step path following-algorithm on a movie graph with 120K > vertici