@Till have you see something like this before? Despite all source tasks
reaching a terminal state on a TM (FAILED) it does not send updates to
the JM for all of them, but only a single one.
On 18/06/2019 12:14, Joshua Fan wrote:
Hi All,
There is a topology of 3 operator, such as, source, parser, and
persist. Occasionally, 5 subtasks of the source encounters exception
and turns to failed, at the same time, one subtask of the parser runs
into exception and turns to failed too. The jobmaster gets a message
of the parser's failed. The jobmaster then try to cancel all the
subtask, most of the subtasks of the three operator turns to canceled
except the 5 subtasks of the source, because the state of the 5 ones
is already FAILED before jobmaster try to cancel it. Then the
jobmaster can not reach a final state but keeps in Failing state
meanwhile the subtask of the source kees in canceling state.
The job run on a flink 1.7 cluster on yarn, and there is only one tm
with 10 slots.
The attached files contains a jm log , tm log and the ui picture.
The exception timestamp is about 2019-06-16 13:42:28.
Yours
Joshua