Re: Best approach to frequently restarting consumer process

2016-12-10 Thread Ewen Cheslack-Postava
Consumer groups aren't going to handle 'let it crash' particularly well (and really any session-based services, but particularly consumer groups since a single failure affects the entire group). That said, 'let it crash' doesn't necessarily have to mean 'don't try to clean up at all'. The consumer

Re: Best approach to frequently restarting consumer process

2016-12-07 Thread Harald Kirsch
With 'restart' I mean a 'let it crash' setup (as promoted by Erlang and Akka, e.g. http://doc.akka.io/docs/akka/snapshot/intro/what-is-akka.html). The consumer gets in trouble due to an OOM or a runaway computation or whatever that we want to preempt somehow. It crashes or gets killed external

Re: Best approach to frequently restarting consumer process

2016-12-06 Thread Gwen Shapira
Can you clarify what you mean by "restart"? If you call consumer.close() and consumer.subscribe() you will definitely trigger a rebalance. It doesn't matter if its "same consumer knocking", we already rebalance when you call consumer.close(). Since we want both consumer.close() and consumer.subsc

Best approach to frequently restarting consumer process

2016-12-06 Thread Harald Kirsch
We have consumer processes which need to restart frequently, say, every 5 minutes. We have 10 of them so we are facing two restarts every minute on average. 1) It seems that nearly every time a consumer restarts the group is rebalanced. Even if the restart takes less than the heartbeat interv