Re: Impact of occasional big pauses in stream processing

2019-02-15 Thread Till Rohrmann
anding is that this should not impact other flink jobs. Is that >> correct? >> >> >> >> Thanks. >> >> >> >> Ajay >> >> >> >> *From: *Andrey Zagrebin >> *Date: *Thursday, February 14, 2019 at 5:09 AM >> *To: *Rong Rong

Re: Impact of occasional big pauses in stream processing

2019-02-14 Thread Rong Rong
derstanding is that this should not impact other flink jobs. Is that > correct? > > > > Thanks. > > > > Ajay > > > > *From: *Andrey Zagrebin > *Date: *Thursday, February 14, 2019 at 5:09 AM > *To: *Rong Rong > *Cc: *"Aggarwal, Ajay" , "

Re: Impact of occasional big pauses in stream processing

2019-02-14 Thread Aggarwal, Ajay
link.apache.org" Subject: Re: Impact of occasional big pauses in stream processing Hi Ajay, Technically, it will immediately block the thread of MyKeyedProcessFunction subtask scheduled to some slot and basically block processing of the key range assigned to this subtask. Practically, I agre

Re: Impact of occasional big pauses in stream processing

2019-02-14 Thread Andrey Zagrebin
Hi Ajay, Technically, it will immediately block the thread of MyKeyedProcessFunction subtask scheduled to some slot and basically block processing of the key range assigned to this subtask. Practically, I agree with Rong's answer. Depending on the topology of your inputStream, it can eventually bl

Re: Impact of occasional big pauses in stream processing

2019-02-13 Thread Rong Rong
Hi Ajay, Flink handles "backpressure" in a graceful way so that it doesn't get affected when your processing pipeline is occasionally slowed down. I think the following articles will help [1,2]. In your specific case: the "KeyBy" operation will re-hash data so they can be reshuffled from all inpu

Impact of occasional big pauses in stream processing

2019-02-13 Thread Aggarwal, Ajay
I was wondering what is the impact if one of the stream operator function occasionally takes too long to process the event. Given the following simple flink job inputStream .KeyBy (“tenantId”) .process ( new MyKeyedProcessFunction()) , if occasionally MyKeyedProcessF