Re: [GraphX] how to set memory configurations to avoid OutOfMemoryError "GC overhead limit exceeded"

[Ankur Dave]

At 2014-09-03 17:58:09 +0200, Yifan LI <iamyifa...@gmail.com> wrote:
> val graph = GraphLoader.edgeListFile(sc, edgesFile, minEdgePartitions = 
> numPartitions).partitionBy(PartitionStrategy.EdgePartition2D).persist(StorageLevel.MEMORY_AND_DISK)
>
> Error: java.lang.UnsupportedOperationException: Cannot change storage level
> of an RDD after it was already assigned a level

You have to pass the StorageLevel to GraphLoader.edgeListFile:

    val graph = GraphLoader.edgeListFile(
      sc, edgesFile, minEdgePartitions = numPartitions,
      edgeStorageLevel = StorageLevel.MEMORY_AND_DISK,
      vertexStorageLevel = StorageLevel.MEMORY_AND_DISK)
      .partitionBy(PartitionStrategy.EdgePartition2D)

Ankur

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org