Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Joel Koshy
Actually from your original mail you do seem to have logs (somewhere - either in a file or stdout). Do you see zookeeper session expirations in there prior to the rebalances? On Fri, Nov 08, 2013 at 04:11:15PM -0500, Ahmed H. wrote: > Thanks for the input. Yes that directory is open for all users

Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Ahmed H.
Thanks for the input. Yes that directory is open for all users (rwx). I don't think that the lack of logging is related to my consumer dying, but it doesn't help when trying to debug when I have no logs. I am struggling to find a reason behind this. I deployed the same code, and same version of K

Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Joel Koshy
Do you have write permissions in /kafka-log4j? Your logs should be going there (at least per your log4j config) - and you may want to use a different log4j config for your consumer so it doesn't collide with the broker's. I doubt the consumer thread dying issue is related to yours - again, logs wo

Re: Kafka client dies after rebalancing attempt

2013-11-08 Thread Ahmed H.
Hello, I am using the beta right now. I'm not sure if it's GC or something else at this point. To be honest I've never really fiddled with any GC settings before. The system can run for as long as a day without failing, or as little as a few hours. The lack of pattern makes it a little harder to

Re: Kafka client dies after rebalancing attempt

2013-11-07 Thread Joel Koshy
Can you see if this applies in your case: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog%3F Also, what version of kafka 0.8 are you using? If not the beta, then what's the git hash? Joel On Thu, Nov 07, 2013 at 02:51:41PM -0500, Ahmed H. wrote