Hi Ankur, Thanks so much for your advice.
But it failed when I tried to set the storage level in constructing a graph. val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = numPartitions).partitionBy(PartitionStrategy.EdgePartition2D).persist(StorageLevel.MEMORY_AND_DISK) Error: java.lang.UnsupportedOperationException: Cannot change storage level of an RDD after it was already assigned a level Is there anyone could give me help? Best, Yifan 2014-08-18 23:52 GMT+02:00 Ankur Dave <ankurd...@gmail.com>: > On Mon, Aug 18, 2014 at 6:29 AM, Yifan LI <iamyifa...@gmail.com> wrote: > > I am testing our application(similar to "personalised page rank" using >> Pregel, and note that each vertex property will need pretty much more space >> to store after new iteration) > > [...] > > But when we ran it on larger graph(e.g. LiveJouranl), it always end at the >> error "GC overhead limit exceeded", even the partitions number is increased >> to 48 from 8. > > > If the graph (including vertex properties) is too large to fit in memory, > you might try allowing it to spill to disk. When constructing the graph, > you can set vertexStorageLevel and edgeStorageLevel to > StorageLevel.MEMORY_AND_DISK. This should allow the algorithm to finish. > > Ankur <http://www.ankurdave.com/> >
