Hey Guozhang, Thanks for reply. I get your point on "hiding" some issues, but I'd prefer to separate the recovery and reporting a failure. Also, I think if simple restart is a possible solution, it shouldn't require implementing it separately or, what's even worse, a manual intervention. Maybe I'll describe my problem then to show you my point of view:
ZK latency spiked for few seconds making ZK effectively dead from consumers' point of view. Then they all reconnected. As I understand, when it happened, it caused rebalancing. Some consumer groups succeeded, but then another spike in latency happened and - as we suspect - it caused rebalancing to fail, because creation of that ZK node failed at some point. Ideally, I'd like to get notified about that problem (rebalancing failed after X retries etc.), so I know there is an issue and I can investigate it, but then I'd like Kafka consumer (or my app) to fallback to restart, which could *possibly* make consumer recover. If not - that's my problem then ;-) In our case it was enough to restart the app to get consumer working again, but - as we didn't know about that behaviour before and we weren't prepared for it - it required manual intervention (on Friday night, which made it even more painful ;> ) which, we believe, wasn't necessary in that case and could have been handled automatically. M. Kind regards, Michał Michalski, michal.michal...@boxever.com On 10 July 2014 23:43, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Michal, > > The rebalance will only be triggered on consumer membership or > topic/partition changes. Once triggered it will try to finish the rebalance > for at most rebalance.max.retries times, i.e. if it fails it will wait for > rebalance.backoff.ms, and then try again until number of retries > exhausted. > When it happens an exception will be thrown and the consumer may be fallen > to a bad state. > > Then reason we did not implement automatic restart upon rebalance failures > is that it may actually "hide" some issues in the systems that actually > caused the rebalance failure. The general design is that if some > exception/errors are not expected like the rebalance failures we will let > it to possibly hault/kill the instance rather than automatically restart > and let it go. > > Guozhang > > > > > On Thu, Jul 10, 2014 at 2:24 AM, Michal Michalski < > michal.michal...@boxever.com> wrote: > > > Hi, > > > > Just wondering - is there any reason why rebalance.max.retries is 4 by > > default? Is there any good reason why I shouldn't expect my consumers to > > keep trying to rebalance for minutes (e.g. 30 retries every 6 seconds), > > rather than seconds (4 retries every 2 seconds by default)? > > > > Also, if my consumer fails to rebalance because of NoNodeException > > (org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > > NoNode for /consumers/is-entity-modified-document-group/ids/<something>) > > wouldn't that make sense to make Kafka restart it automatically once it > > "uses" all the retries attempts? Or recreate the inexistent ZK node > like, I > > believe, it will happen on consumer restart? > > > > I'm asking because that kind of errors seem to be "recoverable" ones, > but - > > if I understand it correctly - with current design they require > > implementing additional mechanisms or manual intervention. > > > > > > Kind regards, > > Michał > > > > > > -- > -- Guozhang >
