Re: Cater to processing longer than max.poll.interval.ms

2018-01-26 Thread Sameer Kumar
I am really not sure of max.poll.interval.ms, do we really need this. Consumer liveless is already ensured by session.timeout.ms/hearbeat.interval.ms. max.poll.interval.ms - by default is 5 mins. session.timeout.ms - 10 secs. if max.poll.interval.ms is not met, then we kill the thread. So, lets sa

Re: Cater to processing longer than max.poll.interval.ms

2018-01-26 Thread Sameer Kumar
Hi, I am talking w.r.t. Kafka 1.0., reducing the poll interval and reducing the number of records polled are always an option. I wanted to explore if there are some other options apart from this and in case of GC pause, both of the above mentioned options will not help. -Sameer. On Fri, Jan 26,

Re: Cater to processing longer than max.poll.interval.ms

2018-01-25 Thread R Krishna
Think, new versions have better ways of doing this. In 0.10.2, because poll() ensure liveness, you can disable auto commits and use consumer pause() to avoid calling poll() (so brokers may ignore max.poll.interval.ms) so those partitions are not assigned to other consumers and also handle ConsumerR

Cater to processing longer than max.poll.interval.ms

2018-01-25 Thread Sameer Kumar
I have a scneario, let say due to GC or any other issue, my consumer takes longer than max.poll.interval.ms to process data, what is the alternative for preventing the consumer to be marked dead and not shun it out of the consumer group. Though the consumer has not died and session.timeout.ms is b