Re: Checkpoint bugs in GraphX

2014-11-10 Thread Xu Lijie
27;m wondering whether the guess is right. 2014-11-11 11:16 GMT+08:00 GuoQiang Li : > I have been trying to fix this bug.‍ > The related PR: > https://github.com/apache/spark/pull/2631‍ > > ------ Original -- > *From: * "Xu Lijie";; > *Dat

Discuss how to do checkpoint more efficently

2014-11-10 Thread Xu Lijie
Hi, all. I want to seek suggestions on how to do checkpoint more efficiently, especially for iterative applications written by GraphX. For iterative applications, the lineage of a job can be very long, which is easy to cause statckoverflow error. A solution is to do checkpoint. However, checkpoin

Checkpoint bugs in GraphX

2014-11-10 Thread Xu Lijie
Hi, all. I'm not sure whether someone has reported this bug: There should be a checkpoint() method in EdgeRDD and VertexRDD as follows: override def checkpoint(): Unit = { partitionsRDD.checkpoint() } Current EdgeRDD and VertexRDD use *RDD.checkpoint()*, which only checkpoint the edges/vertice