Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Jun Rao
It seems you got OutOfMemoryError, which may leave the broker in a bad state. You probably need a larger heap space. Thanks, Jun On Fri, Apr 18, 2014 at 1:58 PM, Alex Demidko wrote: > Last time saw this exception when tried to use rebalance leadership with > kafka-preferred-replica-election.sh

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Alex Demidko
for all partitions from 2 to 2,1 Node 2 starts emit "Closing socket for /Node1 because of error KafkaException: This operation cannot be completed on a complete request", and Node 1 - "EOFException: Received -1 when reading from channel, socket has likely been closed" There is no o

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Guozhang Wang
When you are shutting down the restart node, did you see any warn/error on the leader logs? Guozhang On Fri, Apr 18, 2014 at 1:58 PM, Alex Demidko wrote: > Last time saw this exception when tried to use rebalance leadership with > kafka-preferred-replica-election.sh. That's what got in logs: >

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Alex Demidko
Last time saw this exception when tried to use rebalance leadership with kafka-preferred-replica-election.sh. That's what got in logs: LeaderNode: just kafka.common.KafkaException: This operation cannot be completed on a complete request without any other exceptions. RestartedNode: 2014-04-18 2

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Guozhang Wang
Hello Alex, I think this is a bug on the FetchResponseSend class. Just to confirm, before the kafka.common.KafkaException: This operation cannot be completed on a complete request. do you see other warn/error logs on the current leader? Guozhang On Fri, Apr 18, 2014 at 11:57 AM, Alexander Dem

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Alexander Demidko
Have tried to reproduce this error, and it occurs pretty consistently when node being forcefully shutdown w/o graceful termination. When graceful shutdown was successful no errors occur in a log when the instance was rebooted starts. On Fri, Apr 18, 2014 at 11:17 AM, Alex Demidko wrote: > These

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Alex Demidko
These on alive node: 2014-04-17 21:36:29,276 ERROR [ZkClient-EventThread-15] state.change.logger - Controller 2 epoch 8 encountered error while electing leader for partition [loadtest,143] due to: Preferred replica 1 for partition [loadtest,143] is either not alive or not in the isr. Current le

Re: KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Jun Rao
Any errors from the controller/state-change log? Thanks, Jun On Fri, Apr 18, 2014 at 9:57 AM, Alex Demidko wrote: > Hi, > > I’m performing a producing load test on two node kafka cluster built from > the last 0.8.1 branch sources. I have topic loadtest with replication > factor 2 and 256 parti

KafkaException: This operation cannot be completed on a complete request

2014-04-18 Thread Alex Demidko
Hi, I’m performing a producing load test on two node kafka cluster built from the last 0.8.1 branch sources. I have topic loadtest with replication factor 2 and 256 partitions. Initially both brokers are in ISR and leadership is balanced. When in the middle of the load test one broker was resta