Hi Pierre,
It seems that the community is working on providing a fix with the next
1.11 bugfix release (and for 1.12). You can follow the status of the ticket
here: https://issues.apache.org/jira/browse/FLINK-18934
Best,
Robert
On Thu, Sep 10, 2020 at 11:00 AM Pierre Bedoucha
wrote:
> Hi and
Hi and thank you for this thread,
I'm also experimenting the same valid bug/limitation when connecting
streams.
I had a quick look in the annoucments but couldn't find any more
information: Would it be planned to propagate the Idle stream status to the
operator in the upcoming Flink minor versio
I can only agree with Dawid, who explained it better than me... 😅
Aljoscha
On 31.08.20 12:10, Dawid Wysakowicz wrote:
Hey Arvid,
The problem is that the StreamStatus.IDLE is set on the Task level. It
is not propagated to the operator. Combining of the Watermark for a
TwoInputStreamOperator hap
Hey Arvid,
The problem is that the StreamStatus.IDLE is set on the Task level. It
is not propagated to the operator. Combining of the Watermark for a
TwoInputStreamOperator happens in the AbstractStreamOperator:
   public void processWatermark(Watermark mark) throws Exception {
      if (timeS
Hi Aljoscha,
I don't quite follow your analysis. If both sources are configured with
idleness, they should send a periodic watermark on timeout.
So the code that you posted would receive watermarks on the idle source and
thus advance watermarks periodically.
If an idle source does not emit a wate
Yes, I'm afraid this analysis is correct. The StreamOperator,
AbstractStreamOperator to be specific, computes the combined watermarks
from both inputs here:
https://github.com/apache/flink/blob/f0ed29c06d331892a06ee9bddea4173d6300835d/flink-streaming-java/src/main/java/org/apache/flink/streaming
Hi Kien,
I am afraid this is a valid bug. I am not 100% sure but the way I
understand the code the idleness mechanism applies to input channels,
which means e.g. when multiple parallell instances shuffle its results
to downstream operators.
In case of a two input operator, combining the watermark
Hi all,
We are testing the new Idleness detection feature in Flink 1.11, however,
it does not work as we expected:
When we connect two data streams, of which one is idle, the output
watermark CoProcessOperator does not increase, hence the program cannot
progress.
I've made a small project to illus
