Hi Flink User Group,
I am a data engineer from Thoughtworks and faced an issue recently when
running some flink code . the same code ran fine when run with smaller file
but on increasing the file size it gave this error
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.nextSegment(SimpleCollectingOutputView.java:79)
at
org.apache.flink.runtime.memory.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
at
org.apache.flink.runtime.memory.AbstractPagedOutputView.write(AbstractPagedOutputView.java:190)
at
org.apache.beam.runners.flink.translation.wrappers.DataOutputViewWrapper.write(DataOutputViewWrapper.java:49)
at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786
Somebody mentioned on stackoverflow that this could be a network
buffer issue but Shouldn't the buffers be spilling to disk and making
sure the code doesn't break even it takes more time with larger file?
Can somebody help me with this?
Thanks,
Sambhav Gupta
(DATA ENGINEER)
Thoughtworks
