I actually tried without unpersisting, but given the performance I tryed to add these in order to free the memory. After your anwser I tried to remove them again, but without any change in the execution time...
Looking at the web interface, I can see that the "mapPartitions at GraphImpl.scala:184" step just after the foreach (so I guess computing messages, even if I would have bet on the innerjoin...) are taking ... 2s -> 6s -> 17s -> 48s -> 15m+ (sometimes ends with GC overhead limit exceeded) and have shuffle write of 9MB -> 30MB -> 73MB -> 162MB -> never completed. Is this still related to unpersisting ? I'm mostly a mathematician, and so I'm not well acquainted with the depth of Spark... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-GraphX-pregel-like-with-global-variables-accumulator-broadcast-tp12742p12813.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
