Hi, can you tell me how much memory your job has and how many workers you are running?
>From the trace it seems the internal hash table allocated only 7 MB for the graph data and therefore runs out of memory pretty quickly. Skewed data could also be an issue but with a minimum of 5 pages and a maximum of 8 it seems to be distributed fairly even to the different partitions. Cheers, Robert On Tue, Mar 17, 2015 at 1:25 AM, Mihail Vieru <vi...@informatik.hu-berlin.de > wrote: > And the correct SSSPUnweighted attached. > > > On 17.03.2015 01:23, Mihail Vieru wrote: > >> Hi, >> >> I'm getting the following RuntimeException for an adaptation of the >> SingleSourceShortestPaths example using the Gelly API (see attachment). >> It's been adapted for unweighted graphs having vertices with Long values. >> >> As an input graph I'm using the social network graph (~200MB unpacked) >> from here: https://snap.stanford.edu/data/higgs-twitter.html >> >> For the small SSSPDataUnweighted graph (also attached) it terminates and >> computes the distances correctly. >> >> >> 03/16/2015 17:18:23 IterationHead(WorksetIteration (Vertex-centric >> iteration (org.apache.flink.graph.library.SingleSourceShortestPathsUnwei >> ghted$VertexDistanceUpdater@dca6fe4 | org.apache.flink.graph.library. >> SingleSourceShortestPathsUnweighted$MinDistanceMessenger@6577e8ce)))(2/4) >> switched to FAILED >> java.lang.RuntimeException: Memory ran out. Compaction failed. >> numPartitions: 32 minPartition: 5 maxPartition: 8 number of overflow >> segments: 176 bucketSize: 217 Overall memory: 20316160 Partition memory: >> 7208960 Message: Index: 8, Size: 7 >> at org.apache.flink.runtime.operators.hash. >> CompactingHashTable.insert(CompactingHashTable.java:390) >> at org.apache.flink.runtime.operators.hash.CompactingHashTable. >> buildTable(CompactingHashTable.java:337) >> at org.apache.flink.runtime.iterative.task.IterationHeadPactTask. >> readInitialSolutionSet(IterationHeadPactTask.java:216) >> at org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run( >> IterationHeadPactTask.java:278) >> at org.apache.flink.runtime.operators.RegularPactTask. >> invoke(RegularPactTask.java:362) >> at org.apache.flink.runtime.execution.RuntimeEnvironment. >> run(RuntimeEnvironment.java:205) >> at java.lang.Thread.run(Thread.java:745) >> >> >> Best, >> Mihail >> > >
