Re: Idle stream does not advance watermark in connected stream

[Pierre Bedoucha]

Hi and thank you for this thread,

I'm also experimenting the same valid bug/limitation when connecting
streams. 

I had a quick look in the annoucments but couldn't find any more
information: Would it be planned to propagate the Idle stream status to the
operator in the upcoming Flink minor versions/releases?

If not, would there be a way around it other than emitting beats in the idle
stream?

Best,
Pierre


Aljoscha Krettek wrote
> I can only agree with Dawid, who explained it better than me... 😅
> 
> Aljoscha
> 
> On 31.08.20 12:10, Dawid Wysakowicz wrote:
>> Hey Arvid,
>> 
>> The problem is that the StreamStatus.IDLE is set on the Task level. It
>> is not propagated to the operator. Combining of the Watermark for a
>> TwoInputStreamOperator happens in the AbstractStreamOperator:
>> 
>>      public void processWatermark(Watermark mark) throws Exception {
>>          if (timeServiceManager != null) {
>>              timeServiceManager.advanceWatermark(mark);
>>          }
>>          output.emitWatermark(mark);
>>      }
>> 
>>      public void processWatermark1(Watermark mark) throws Exception {
>>          input1Watermark = mark.getTimestamp();
>>          long newMin = Math.min(input1Watermark, input2Watermark);
>>          if (newMin > combinedWatermark) {
>>              combinedWatermark = newMin;
>>              processWatermark(new Watermark(combinedWatermark));
>>          }
>>      }
>> 
>>      public void processWatermark2(Watermark mark) throws Exception {
>>          input2Watermark = mark.getTimestamp();
>>          long newMin = Math.min(input1Watermark, input2Watermark);
>>          if (newMin > combinedWatermark) {
>>              combinedWatermark = newMin;
>>              processWatermark(new Watermark(combinedWatermark));
>>          }
>>      }
>> 
>> There we do not know that e.g. the whole input 1 is idle. Therefore if
>> we do not receive any Watermarks from it (it became IDLE) we do not
>> progress the Watermark starting from any two input operator. We are
>> missing similar handling of the IDLE status from the task level which
>> works well for one input operators and multiple parallel upstream
>> instances.
>> 
>> Best,
>> 
>> Dawid
>> 
>>





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/