Re: Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-11 Thread Ashish Pokharel
Hi Gordon, Any further thoughts on this? I forgot to mention I am using Flink 1.3.2 and our Kafka is 0.10. We are in the process of upgrading Kafka but won’t be in Prod for at least couple of months. Thanks, Ashish > On Nov 8, 2017, at 9:35 PM, Ashish Pokharel wrote: > > Hi Grodon, > > Than

Re: Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-08 Thread Ashish Pokharel
Hi Grodon, Thanks for your responses. It definitely makes sense. I could pull this stack from the logs, entire log itself is pretty big - let me know if some samples before/after this may help. TimerException{org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could

Re: Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-08 Thread Tzu-Li (Gordon) Tai
Hi Ashish, From your description I do not yet have much of an idea of what may be happening. However, some of your observations seems reasonable. I’ll go through them one by one: I did try to modify request.timeout.ms, linger.ms etc to help with the issue if it were caused by a sudden burst of

Re: Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-07 Thread ashish pok
Thanks Fabian. I am seeing thia consistently and can definitely use some help. I have plenty of graphana views I can share if that helps :) Sent from Yahoo Mail on Android On Tue, Nov 7, 2017 at 3:54 AM, Fabian Hueske wrote: Hi Ashish, Gordon (in CC) might be able to help you. Cheers, Fab

Re: Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-07 Thread Fabian Hueske
Hi Ashish, Gordon (in CC) might be able to help you. Cheers, Fabian 2017-11-05 16:24 GMT+01:00 Ashish Pokharel : > All, > > I am starting to notice a strange behavior in a particular streaming app. > I initially thought it was a Producer issue as I was seeing timeout > exceptions (records expir

Kafka Consumer fetch-size/rate and Producer queue timeout

2017-11-05 Thread Ashish Pokharel
All, I am starting to notice a strange behavior in a particular streaming app. I initially thought it was a Producer issue as I was seeing timeout exceptions (records expiring in queue. I did try to modify request.timeout.ms, linger.ms etc to help with the issue if it were caused by a sudden bu