At 2014-08-26 01:20:09 -0700, BertrandR <bertrand.rondepierre...@gmail.com> wrote: > I actually tried without unpersisting, but given the performance I tryed to > add these in order to free the memory. After your anwser I tried to remove > them again, but without any change in the execution time...
This is probably a related issue: in Spark you have to explicitly cache any dataset that you use more than once. Otherwise it will be recomputed each time it's used, which can cause an exponential slowdown for certain dependency structures. To be safe, you could start by caching g, msg, and newVerts every time they are set. Ankur --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
