[
https://issues.apache.org/jira/browse/FLINK-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186823#comment-17186823
]
Kenneth William Krugler commented on FLINK-18934:
-------------------------------------------------
On the list, [~dwysakowicz] said:
{quote}Hi Kien,
I am afraid this is a valid bug. I am not 100% sure but the way I
understand the code the idleness mechanism applies to input channels,
which means e.g. when multiple parallell instances shuffle its results
to downstream operators.
In case of a two input operator, combining the watermark of two
different upstream operators happens inside of the operator itself.
There we do not have the idleness status. We do not have a status that a
whole upstream operator became idle. That's definitely a bug/limitation.
I'm also cc'ing Aljoscha who could maybe confirm my analysis.
Best,
Dawid{quote}
And [~aljoscha] added:
{quote}Yes, I'm afraid this analysis is correct. The StreamOperator,
AbstractStreamOperator to be specific, computes the combined watermarks from
both inputs here:
https://github.com/apache/flink/blob/f0ed29c06d331892a06ee9bddea4173d6300835d/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java#L573.
The operator layer is not aware of idleness so it will never notice. The
idleness only works on the level of inputs but is never forwarded to an
operator itself.
To fix this we would have to also make operators aware of idleness such that
they can take this into account when computing the combined output watermark.
Best,
Aljoscha{quote}
> Idle stream does not advance watermark in connected stream
> ----------------------------------------------------------
>
> Key: FLINK-18934
> URL: https://issues.apache.org/jira/browse/FLINK-18934
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
> Affects Versions: 1.11.1
> Reporter: Truong Duc Kien
> Priority: Major
>
> Per Flink documents, when a stream is idle, it will allow watermarks of
> downstream operator to advance. However, when I connect an active data stream
> with an idle data stream, the output watermark of the CoProcessOperator does
> not increase.
> Here's a small test that reproduces the problem.
> https://github.com/kien-truong/flink-idleness-testing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
