I think this is not the root cause of job failure, this task is caused by other tasks failing. You can check the log of the first failed task.
2017-03-10 12:25 GMT+08:00 Govindarajan Srinivasaraghavan < govindragh...@gmail.com>: > Hi All, > > I see the below error after running my streaming job for a while and when > the load increases. After a while the task manager becomes completely dead > and the job keeps on restarting. > > Also when I checked if there is an back pressure in the UI, it kept on > saying sampling in progress and no results were displayed. Is there an API > which can provide the back pressure details? > > 2017-03-10 01:40:58,793 WARN > org.apache.flink.streaming.api.operators.AbstractStreamOperator > - Error while emitting latency marker. > org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: > Could not forward element to next operator > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > ChainingOutput.emitLatencyMarker(OperatorChain.java:426) > at org.apache.flink.streaming.api.operators. > AbstractStreamOperator$CountingOutput.emitLatencyMarker( > AbstractStreamOperator.java:848) > at org.apache.flink.streaming.api.operators.StreamSource$ > LatencyMarksEmitter$1.onProcessingTime(StreamSource.java:152) > at org.apache.flink.streaming.runtime.tasks. > SystemProcessingTimeService$RepeatedTriggerTask.run( > SystemProcessingTimeService.java:256) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset( > FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.RuntimeException > at org.apache.flink.streaming.runtime.io.RecordWriterOutput. > emitLatencyMarker(RecordWriterOutput.java:117) > at org.apache.flink.streaming.api.operators. > AbstractStreamOperator$CountingOutput.emitLatencyMarker( > AbstractStreamOperator.java:848) > at org.apache.flink.streaming.api.operators. > AbstractStreamOperator.reportOrForwardLatencyMarker( > AbstractStreamOperator.java:708) > at org.apache.flink.streaming.api.operators. > AbstractStreamOperator.processLatencyMarker(AbstractStreamOperator.java: > 690) > at org.apache.flink.streaming.runtime.tasks.OperatorChain$ > ChainingOutput.emitLatencyMarker(OperatorChain.java:423) > ... 10 more > Caused by: java.lang.InterruptedException > at java.lang.Object.wait(Native Method) > at org.apache.flink.runtime.io.network.buffer. > LocalBufferPool.requestBuffer(LocalBufferPool.java:168) > at org.apache.flink.runtime.io.network.buffer.LocalBufferPool. > requestBufferBlocking(LocalBufferPool.java:138) > at org.apache.flink.runtime.io.network.api.writer. > RecordWriter.sendToTarget(RecordWriter.java:132) > at org.apache.flink.runtime.io.network.api.writer. > RecordWriter.randomEmit(RecordWriter.java:107) > at org.apache.flink.streaming.runtime.io.StreamRecordWriter. > randomEmit(StreamRecordWriter.java:104) > at org.apache.flink.streaming.runtime.io.RecordWriterOutput. > emitLatencyMarker(RecordWriterOutput.java:114) > ... 14 more > > >
