+1 on the idea. We have a ticket about making all the blocking call have a timeout in KafkaConsumer. The implementation could be a little tricky as Ewen mentioned. But for close it is probably a simpler case because in the worst case the consumer will just stop polling and heartbeating and eventually got kicked out of the group. It is not ideal but maybe less worrisome.
On Thu, Dec 15, 2016 at 3:08 PM, Rajini Sivaram <rsiva...@pivotal.io> wrote: > Ewen, > > Thank you, I will try to prototype a solution early next week to get a > better understanding of how invasive the changes are. > > > > On Thu, Dec 15, 2016 at 9:35 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Rajini, > > > > Thanks for this KIP, I'd definitely like to see this. Connect has had a > > long-standing TODO around stopping sink tasks where we can't properly > > manage the rebalance process (which involves stopping consumers) because > we > > lack a timeout here. Not a huge problem in practice, but would be nice to > > fix. And I know I've seen at least a half dozen requests for this (and > > other timeouts to be respected in the consumer) on the mailing list. > > > > My only concern is that this could be a pretty substantial change to the > > consumer code. We've had TODO items in the code since Jay wrote the first > > version that say something about avoiding infinite looping and waiting > > indefinitely on group membership. If we implement this, I'd hope to get > it > > committed in the early part of the release cycle so we have plenty of > time > > to stabilize + bug fix. > > > > -Ewen > > > > On Thu, Dec 15, 2016 at 5:32 AM, Rajini Sivaram <rsiva...@pivotal.io> > > wrote: > > > > > Hi all, > > > > > > I have just created KIP-102 to add a new close method for consumers > with > > a > > > timeout parameter, making Consumer consistent with Producer: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 102+-+Add+close+with+timeout+for+consumers > > > > > > Comments and suggestions are welcome. > > > > > > Thank you... > > > > > > Regards, > > > > > > Rajini > > > > > >
