Hi Ajay, Yes, Andrey is right. I was actually missing the first basic but important point: If your process function is stuck, it will immediately block that thread. >From your description, what it sounds like is that not all the messages you consume from kafka actually triggers the processing logic. There should be plenty of way to avoid over provisioning your job just to satisfy your peak traffic: for example what Andrey suggested, using a Async RPC call to some other resource for the heavy computation; or split it into a filter (which removes non-actionable messages) and the actual process (where you can use a higher parallelism to reduce chances of stuck).
Regarding the second question, I am not expert but my understanding is that there should be isolations between Flink jobs you run on that session cluster. e.g. one job's backpressure will not affect other jobs' consumer. I've CCed Till who might be able to better answer your question. -- Rong On Thu, Feb 14, 2019 at 8:24 AM Aggarwal, Ajay <ajay.aggar...@netapp.com> wrote: > Thank you Rong and Andrey. The blog and your explanation was very useful. > > > > In my use case, source stream (kafka based) contains messages that capture > some “work” that needs to be done for a tenant. It’s a multi-tenant source > stream. I need to queue up (and execute) this work per tenant in the order > in which it was produced. And flink provides this ordered queuing per > tenant very elegantly. Now the only thing is that executing this “work” > could be expensive in terms of compute/memory/time. Furthermore per tenant > there is a constraint of doing this work serially. Hence this question. I > believe if our flink cluster has enough resources, it should work. > > > > But this leads to another related question. If there are multiple flink > jobs sharing the same flink cluster and one of those jobs sees the spike > such that back pressure builds up all the way to the source, will that > impact other jobs as well? Is a task slot shared by multiple jobs? If not, > my understanding is that this should not impact other flink jobs. Is that > correct? > > > > Thanks. > > > > Ajay > > > > *From: *Andrey Zagrebin <and...@ververica.com> > *Date: *Thursday, February 14, 2019 at 5:09 AM > *To: *Rong Rong <walter...@gmail.com> > *Cc: *"Aggarwal, Ajay" <ajay.aggar...@netapp.com>, "user@flink.apache.org" > <user@flink.apache.org> > *Subject: *Re: Impact of occasional big pauses in stream processing > > > > Hi Ajay, > > > > Technically, it will immediately block the thread of > MyKeyedProcessFunction subtask scheduled to some slot and basically block > processing of the key range assigned to this subtask. > Practically, I agree with Rong's answer. Depending on the topology of your > inputStream, it can eventually block a lot of stuff. > In general, I think, it is not recommended to perform blocking operations > in process record functions. You could consider AsyncIO [1] to unblock the > task thread. > > Best, > > Andrey > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/asyncio.html > > > > On Thu, Feb 14, 2019 at 6:03 AM Rong Rong <walter...@gmail.com> wrote: > > Hi Ajay, > > > > Flink handles "backpressure" in a graceful way so that it doesn't get > affected when your processing pipeline is occasionally slowed down. > > I think the following articles will help [1,2]. > > > > In your specific case: the "KeyBy" operation will re-hash data so they can > be reshuffled from all input consumers to all your process operators (in > this case the MyKeyedProcessFunction). If one of the process operator is > backpressured, it will back track all the way to the source. > > So, my understanding is that: since there's the reshuffling, if one of the > process function is backpressured, it will potentially affect all the > source operators. > > > > Thanks, > > Rong > > > > [1] https://www.ververica.com/blog/how-flink-handles-backpressure > > [2] > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/back_pressure.html > > > > On Wed, Feb 13, 2019 at 8:50 AM Aggarwal, Ajay <ajay.aggar...@netapp.com> > wrote: > > I was wondering what is the impact if one of the stream operator function > occasionally takes too long to process the event. Given the following > simple flink job > > > > inputStream > > .KeyBy (“tenantId”) > > .process ( new MyKeyedProcessFunction()) > > > > , if occasionally MyKeyedProcessFunction takes too long (say ~5-10 > minutes) to process an incoming element, what is the impact on overall > pipeline? Is the impact limited to > > 1. Specific key for which MyKeyedProcessFunction is currently taking > too long to process an element, or > 2. Specific Taskslot, where MyKeyedProcessFunction is currently taking > too long to process an element, i.e. impacting multiple keys, or > 3. Entire inputstream ? > > > > Also what is the built in resiliency in these cases? Is there a concept of > timeout for each operator function? > > > > Ajay > >
