Re: Zookeeper reconnect failed due to 'state changed (Expired)'

2014-07-02 Thread Jun Rao
Are you on Linux? We have seen this pattern (user/sys time low and real time high in GC time) before. In our case, the problem was due to disk I/Os. When there are lots of dirty pages (in our case, this is caused by log4j logging), Linux can draft user threads (in this case GC threads) to flush the

Re: Zookeeper reconnect failed due to 'state changed (Expired)'

2014-07-02 Thread Andrew Otto
Hi again! I've been having this issue consistently since I first started this thread, but it was happening infrequently enough for me to brush it aside and just run an election to rebalance brokers. I recently expanded (and reinstalled) our Kafka cluster so that it now has 4 brokers with a defaul

Re: Kafka producer performance test sending 0x0 byte messages

2014-07-02 Thread Bert Corderman
Jun, let me see if I can fix first and then will submit back. Daniel, I was looking at the code some more and was thinking this might work https://github.com/apache/kafka/blob/0.8.1/perf/src/main/scala/kafka/perf/ProducerPerformance.scala on line 246 instead of looping to create messages I co

Re: Failed to send messages after 3 tries

2014-07-02 Thread Jun Rao
There are a couple of things you can try. 1. See if broker 1 and 2 are indeed registered in ZK (see Broker registration info in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper ). 2. Does restarting broker 1 and 2 solve the issue? Thanks, Jun On Tue, Jul 1