Re: Kafka connection loss with high volume of messages

2014-06-26 Thread Neha Narkhede
11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient] (pool-5-thread-1-EventThread) zookeeper state changed (Disconnected) 11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient] (clojure-agent-send-off-pool- 6-EventThread) zookeeper state changed (Disconnected) I wonder why your consumer disconnects from z

Re: Kafka connection loss with high volume of messages

2014-06-26 Thread Ahmed H.
I dug some more and it seems like before these errors show up, I see a few Zookeeper warnings, followed by Kafka errors. 11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient] (pool-5-thread-1-EventThread) zookeeper state changed (Disconnected) 11:57:26,897 INFO [org.I0Itec.zkclient.ZkClient] (clojure

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Neha Narkhede
If rebalance succeeded, then those error messages are harmless. Though I agree we shouldn't log those in the first place. On Wed, Jun 25, 2014 at 2:12 PM, Ahmed H. wrote: > Unfortunately I do not have the logs on hand anymore, they were cleared > already. > > With that said, I do recall seeing

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Ahmed H.
Unfortunately I do not have the logs on hand anymore, they were cleared already. With that said, I do recall seeing some rebalancing. It attempts to rebalance a few times and eventually succeeds. In the past, I have had cases where it tries rebalancing 4 times and gives up because it reached it's

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Neha Narkhede
Do you see something like "begin rebalancing consumer" in your consumer logs? Could you send around the full log4j of the consumer? On Wed, Jun 25, 2014 at 8:19 AM, Ahmed H. wrote: > Are you referring to the zookeeper logs? If so, I am seeing a lot of those: > > 2014-06-25 11:15:02 NIOServerCnx

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Ahmed H.
Are you referring to the zookeeper logs? If so, I am seeing a lot of those: 2014-06-25 11:15:02 NIOServerCnxn [WARN] caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x146958701700371, likely client has closed socket at org.apache.zookeeper

Re: Kafka connection loss with high volume of messages

2014-06-25 Thread Guozhang Wang
Hello Ahmed, Did you see any exceptions on the broker logs? Guozhang On Wed, Jun 25, 2014 at 7:47 AM, Ahmed H. wrote: > Hello All, > > I am seeing this issue very frequently when running a high volume of > messages through Kafka. It starts off well, and it can go on for minutes > that way, bu