Hi, I managed to reproduce the behavior and as far as I can tell it seems to be a problem with the memory allocation.
I have filed a bug report in JIRA to get the attention of somebody who knows the runtime better than I do. https://issues.apache.org/jira/browse/FLINK-1734 Cheers, Robert On Tue, Mar 17, 2015 at 3:52 PM, Mihail Vieru <vi...@informatik.hu-berlin.de > wrote: > Hi Robert, > > thank you for your reply. > > I'm starting the job from the Scala IDE. So only one JobManager and one > TaskManager in the same JVM. > I've doubled the memory in the eclipse.ini settings but I still get the > Exception. > > -vmargs > -Xmx2048m > -Xms100m > -XX:MaxPermSize=512m > > Best, > Mihail > > > On 17.03.2015 10:11, Robert Waury wrote: > > Hi, > > can you tell me how much memory your job has and how many workers you are > running? > > From the trace it seems the internal hash table allocated only 7 MB for > the graph data and therefore runs out of memory pretty quickly. > > Skewed data could also be an issue but with a minimum of 5 pages and a > maximum of 8 it seems to be distributed fairly even to the different > partitions. > > Cheers, > Robert > > On Tue, Mar 17, 2015 at 1:25 AM, Mihail Vieru < > vi...@informatik.hu-berlin.de> wrote: > >> And the correct SSSPUnweighted attached. >> >> >> On 17.03.2015 01:23, Mihail Vieru wrote: >> >>> Hi, >>> >>> I'm getting the following RuntimeException for an adaptation of the >>> SingleSourceShortestPaths example using the Gelly API (see attachment). >>> It's been adapted for unweighted graphs having vertices with Long values. >>> >>> As an input graph I'm using the social network graph (~200MB unpacked) >>> from here: https://snap.stanford.edu/data/higgs-twitter.html >>> >>> For the small SSSPDataUnweighted graph (also attached) it terminates and >>> computes the distances correctly. >>> >>> >>> 03/16/2015 17:18:23 IterationHead(WorksetIteration (Vertex-centric >>> iteration >>> (org.apache.flink.graph.library.SingleSourceShortestPathsUnweighted$VertexDistanceUpdater@dca6fe4 >>> | >>> org.apache.flink.graph.library.SingleSourceShortestPathsUnweighted$MinDistanceMessenger@6577e8ce)))(2/4) >>> switched to FAILED >>> java.lang.RuntimeException: Memory ran out. Compaction failed. >>> numPartitions: 32 minPartition: 5 maxPartition: 8 number of overflow >>> segments: 176 bucketSize: 217 Overall memory: 20316160 Partition memory: >>> 7208960 Message: Index: 8, Size: 7 >>> at >>> org.apache.flink.runtime.operators.hash.CompactingHashTable.insert(CompactingHashTable.java:390) >>> at >>> org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTable(CompactingHashTable.java:337) >>> at >>> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:216) >>> at >>> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:278) >>> at >>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362) >>> at >>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:205) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> Best, >>> Mihail >>> >> >> > >
