Hi! After a quick look over the code, this seems like a bug. One cornercase of the overflow handling code does not check for the "running out of memory" condition.
I would like to wait if Robert Waury has some ideas about that, he is the one most familiar with the code. I would guess, though, that you should be able to work around that by either setting the solution set as "unmanaged", or by slightly changing the memory configuration. It seems a rare cornercase, only if the memory runs out in a very special situation. You may be able to avoid running into it when using slightly more or less memory. Greetings, Stephan On Fri, Apr 17, 2015 at 3:59 PM, Stefan Bunk <[email protected]> wrote: > Hi Squirrels, > > I have some trouble with a delta-iteration transitive closure program [1]. > When I run the program, I get the following error: > > java.io.EOFException > at > org.apache.flink.runtime.operators.hash.InMemoryPartition$WriteView.nextSegment(InMemoryPartition.java:333) > at > org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140) > at > org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeByte(AbstractPagedOutputView.java:223) > at > org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeLong(AbstractPagedOutputView.java:291) > at > org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeDouble(AbstractPagedOutputView.java:307) > at > org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:62) > at > org.apache.flink.api.common.typeutils.base.DoubleSerializer.serialize(DoubleSerializer.java:26) > at > org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:89) > at > org.apache.flink.api.scala.typeutils.CaseClassSerializer.serialize(CaseClassSerializer.scala:29) > at > org.apache.flink.runtime.operators.hash.InMemoryPartition.appendRecord(InMemoryPartition.java:219) > at > org.apache.flink.runtime.operators.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:536) > at > org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTableWithUniqueKey(CompactingHashTable.java:347) > at > org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:209) > at > org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:270) > at > org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362) > at > org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217) > at java.lang.Thread.run(Thread.java:745) > > Both input files have been generated and written to HDFS by Flink jobs. I > already ran the Flink program that generated them several times: the error > persists. > You can find the logs at [2] and [3]. > > I am using the 0.9.0-milestone-1 release. > > Best, > Stefan > > > [1] Flink job: https://gist.github.com/knub/0bfec859a563009c1d57 > [2] Job manager logs: https://gist.github.com/knub/01e3a4b0edb8cde66ff4 > [3] One task manager's logs: > https://gist.github.com/knub/8f2f953da95c8d7adefc >
