Re: Checkpointing in Iterative Graph Computation

2015-10-10 Thread Robineast
One other thought - you need to call SparkContext.setCheckpointDir otherwise nothing will happen - Robin East Spark GraphX in Action Michael Malak and Robin East Manning Publications Co. http://www.manning.com/books/spark-graphx-in-action -- View this message in context: http://apache-s

Re: Checkpointing in Iterative Graph Computation

2015-10-10 Thread Robineast
You need to checkpoint before you materialize. You'll find you probably only want to checkpoint every 100 or so iterations otherwise the checkpointing will slow down your application excessively - Robin East Spark GraphX in Action Michael Malak and Robin East Manning Publications Co. http